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EXECUTIVE  SUMMARY 


Introduction 

The  United  States  Coast  Guard  (USCG)  has  introduced  significant  changes  in  the  requirements 
for  the  training  and  qualifications  of  mariners.  These  changes  occur  in  the  context  of  heightened 
concern  for  the  safety  of  navigation,  increasingly  challenging  shipboard  conditions,  and  changing 
international  agreements,  all  of  which  have  resulted  in  greater  demands  on  the  competence  of 
ships’  crews.  The  new  requirements  emphasize  the  practical  demonstration  and  critical 
assessment  of  mariner  performance  to  ensure  competence  for  licensing.  To  provide  technical 
support  to  the  USCG  National  Maritime  Center  (NMC)  in  the  implementation  of  these  new 
requirements,  the  USCG  Research  and  Development  Center  (R&DC)  investigated  some  of  the 
central  issues  in  assessment.  Project  findings  provide  model  processes  for  the  development  of 
rigorous  and  practical  performance-based  assessment  procedures  by  industry,  and  guidelines  for 
the  critical  review  of  these  procedures  by  the  USCG  or  third-party  reviewers. 

Objectives  and  Technical  Approach 

The  primary  objective  was  to  develop  an  approach  for  evaluating  a  simulator’s  capability  to 
support  specific,  performance-based  assessment  procedures.  The  basic  assumption  of  the 
approach  is  that  performance-based  objectives  for  the  assessment  of  specific  mariner 
proficiencies  should  serve  as  the  technical  foundation  for  simulator  evaluations.  That  is,  the 
performance  that  an  assessment  candidate  must  demonstrate,  and  the  conditions  required  for 
demonstrating  that  performance,  should  provide  the  basis  for  specifying  the  required 
characteristics  of  a  simulator  to  support  a  valid  assessment.  This  report  documents  a  step-by-step 
method  for  determining  the  required  simulator  characteristics  and  for  preparing  a  protocol  to  be 
used  for  evaluating  a  simulator.  The  approach  developed  is  based  on:  (1)  mariner  performance 
requirements;  (2)  assessment  conditions  required  for  demonstrating  performance;  and  (3) 
operational  requirements  for  the  shipboard  equipment  used  by  the  mariner.  This  approach  can  be 
applied  to  simulators  of  varying  capabilities  and  cost.  This  will  help  to  identify  the  most  cost- 
effective  simulator  for  a  valid  assessment  of  mariner  performance. 

To  test  the  approach,  an  assessment  of  Automatic  Radar  Plotting  Aid  (ARPA)  operation  was 
used  as  a  case  study.  The  ARPA  operation  was  chosen  because  it  is  one  of  the  proficiencies 
mandated  by  the  STCW  Code.  STCW-defined  mariner  performance  assessment  objectives  for 
ARPA  operation  were  taken  as  a  starting  point,  and  the  conditions  for  their  assessment  were  used 
as  a  basis  for  determining  the  required  simulator  characteristics.  These  requirements  were 
incorporated  into  a  simulator  evaluation  protocol,  which  was  applied  to  two  commercially- 
available  desktop  ARPA  simulators. 

Conclusions 

The  project  demonstrated  the  feasibility  of  the  approach  to  simulator  evaluation.  The  ARPA 
simulator  case  study  successfully  identified  important  differences  in  the  capabilities  of  two 
different  desktop  simulators  to  support  performance-based  assessments.  Further,  the  present 
evaluation  protocol  explicitly  incorporated  standards  for  simulators  established  by  the  IMO 
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STCW  Code.  Thus,  the  more  general  simulator  evaluation  method  can  be  fully  consistent  with 
these  IMO  standards. 

The  present  application  of  the  simulator  evaluation  method  was  limited  to  PC-based  ARPA 
simulators.  However,  the  method  is  generic  and  has  a  broad  range  of  potential  applications,  such 
as:  more  complex  ARPA  simulators;  other  bridge  and  engine  room  equipment  simulators;  other 
maritime  simulators  (such  as  vessel  traffic  system  simulators);  and  simulators  used  in  other 
industries  (e.g.,  flight  simulators,  driving  simulators,  power  plant  control  room  simulators). 
Although  the  present  application  focused  on  the  use  of  simulators  for  performance  assessment, 
the  evaluation  of  simulators  for  use  in  training  applications  would  be  similar.  Once  objectives 
are  established  for  a  training  program,  simulator  evaluation  criteria  could  be  developed  based  on 
training  requirements. 

There  is  a  broad  range  of  potential  users  of  the  evaluation  method.  Training  institutions  (e.g., 
maritime  academies,  colleges,  and  commercial  training  centers)  could  use  the  method  for 
selecting  cost-effective  simulators  to  meet  their  needs.  The  USCG,  other  regulatory  agencies,  or 
third-party  reviewers  could  adapt  the  method  in  developing  standardized  evaluation  procedures 
for  different  types  of  simulators.  Simulator  manufacturers  could  use  the  method  to  identify  the 
features  and  capabilities  needed  in  new  simulators  and  in  upgrades  to  existing  simulators. 

Recommendations 

The  following  actions  by  the  USCG,  maritime  academies  and  other  training  institutions  will 
make  the  most  effective  use  of  the  research  findings: 

•  The  USCG  should  make  the  current  methodology  widely  available  to  the  maritime 
community  and  encourage  its  inclusion  in  performance-based  assessment  courses  or  in  “train- 
the-trainer”  courses. 

•  The  simulator  evaluation  method  documented  in  this  report  should  be  applied  to  a  wide  range 
of  simulators  so  as  to  create  a  library  of  simulator  evaluation  protocols.  The  USCG  should 
encourage  the  maritime  academies  and  other  appropriate  institutions  to  apply  the 
methodology  to  other  types  of  simulators,  share  general  lessons  learned,  model  protocols  for 
other  types  of  simulators,  and  share  actual  results  of  these  evaluations. 

•  The  USCG  should  develop,  or  encourage  appropriate  institutions  to  develop,  standardized 
evaluation  procedures  for  various  types  of  simulators.  These  procedures  could  include 
standard  scenarios  and  conditions,  as  well  as  guidelines  and  cut-off  scores  for  accepting  or 
not  accepting  a  simulator  or  a  course  based  on  it. 

•  The  USCG  should  make  the  ARPA  simulator  evaluation  protocol  widely  available  and 
consider  requiring  its  use  as  a  standard  evaluation  for  ARPA  simulators  used  in  training 
courses. 
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INTRODUCTION 


In  this  report  we  describe  a  method  for  evaluating  the  capability  of  simulators  to  support  mariner 
performance  assessments.  The  method  is  illustrated  with  the  development  of  an  evaluation 
protocol  and  its  application  in  evaluating  two  Automatic  Radar  Plotting  Aid  (ARPA)  simulators. 
The  simulator  evaluation  method  described  in  this  report  may  be  applied  to  other  bridge  and 
engine  room  simulators  used  in  mariner  training  and  assessment,  as  well  as  to  simulators 
designed  for  use  in  other  industries. 

The  research  leading  up  to  this  report  was  part  of  a  broader  research  project  conducted  for  the 
United  States  Coast  Guard  (USCG)  National  Maritime  Center  (NMC)  by  the  USCG  Research 
and  Development  Center  (R&DC).  The  overall  objective  of  the  research  project  is  to  establish  a 
methodology  for  conducting  assessments  of  mariner  proficiencies  using  practical  demonstration. 
As  part  of  this  broader  project,  a  rigorous  method  for  developing  performance-based  assessment 
procedures  was  developed  and  documented  (McCallum,  Forsythe,  Smith,  Nunnenkamp,  & 
Sandberg,  2000)  hereafter  referred  to  as  “McCallum  et  al.  (2000).”  This  assessment 
development  method  provided  much  of  the  theoretical  foundation  for  the  present  effort. 

Role  of  Simulators  in  Performance-Based  Assessments 

The  International  Maritime  Organization  (IMO)  Seafarers’  Training,  Certification  and 
Watchkeeping  (STCW)  Code  indicates  that  mariner  proficiency  should  be  assessed  by  practical 
demonstration  (IMO,  1996).  This  means  that  to  be  considered  proficient,  a  mariner  must  be  able 
to  perform  a  variety  of  shipboard  operations  safely  and  effectively  in  a  real  or  operationally 
realistic  setting  to  the  satisfaction  of  an  expert.  As  discussed  in  McCallum  et  al.  (2000),  these 
demonstrations  of  performance  should  be  assessed  using  an  assessment  procedure  based  on  a 
well-defined  set  of  performance  objectives.  This  type  of  assessment  is  called  performance-based 
assessment. 

Performance-based  assessment  can  occur  aboard  a  ship,  using  a  simulator,  or  in  a  classroom 
setting.  Marine  simulators  frequently  are  used  for  performance-based  assessment  because  they 
provide  a  safe  and  convenient  alternative  to  shipboard  assessment.  Using  a  simulator,  a  mariner 
can  be  assessed  under  controlled  conditions  on  a  wide  variety  of  shipboard  operations.  These 
operations  include  emergency  or  hazardous  situations  that  often  cannot  be  replicated  for 
shipboard  assessment  because  of  safety  and  cost  considerations.  Additionally,  simulators  allow  a 
mariner  to  practice  exercises  repeatedly,  and  many  simulators  provide  performance  feedback. 

All  of  these  characteristics  are  useful  for  training,  as  well  as  assessment,  because  they  allow 
mariners  to  be  introduced  to  new  tasks  without  any  danger  to  themselves  or  their  coworkers 
(Goldstein,  1993). 

There  is  a  broad  range  of  commercially  available  marine  simulators  that  can  be  used  to  support 
any  given  mariner  assessment.  These  simulators  range  from  those  designed  to  provide  highly 
realistic  operational  environments,  controls,  and  displays  to  those  that  are  designed  to  represent  a 
facsimile  of  some  limited  portion  of  operational  information.  Examples  of  more  complex 
simulators  are  full-mission  bridge  simulators  and  ARPA  simulators  featuring  high-fidelity  inputs 
into  actual  operational  equipment.  Examples  of  less  technically  complex  simulators  are  personal 
computer-  (PC)  based  simulators  featuring  standard  PC  displays  and  controls  in  place  of  actual 
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operational  equipment.  Obviously,  a  simulator  that  is  a  full-scale  mock-up  of  the  operational 
setting  is  likely  to  have  many  of  the  shipboard  equipment  features  and  functions  necessary  to 
conduct  a  valid  mariner  assessment.  However,  with  the  advances  in  PC  processing  power  and 
the  incorporation  of  PC  processing  in  shipboard  equipment,  the  advantages  of  the  more  complex 
and  elaborate  simulators  are  becoming  less  pronounced.  When  these  factors  are  considered  in 
conjunction  with  the  affordability  of  PC-based  simulators,  it  becomes  evident  that  the  capability 
of  PC  simulators  to  support  mariner  assessment  must  be  considered  seriously. 

The  amended  STCW  Code  specifically  requires  that  simulators  be  used  to  assess  mariner 
proficiency  in  radar  and  ARPA  operation.  The  STCW  Code  also  specifies  that  simulators  may  be 
used  for  demonstrating  proficiency  in  other  areas,  such  as  maintaining  a  safe  engineering  watch 
or  monitoring  the  loading  and  unloading  of  cargo.  In  addition,  other  competencies  not 
emphasized  in  the  STCW  amendments,  such  as  operating  a  Global  Maritime  Distress  and  Safety 
System  (GMDSS)  or  an  Electronic  Chart  Display  and  Information  System  (ECDIS),  also  can  be 
trained  and  assessed  using  simulators.  Marine  simulators  varying  significantly  in  fidelity  and 
cost  are  available  for  each  of  these  areas. 

ARPA  simulators  provide  a  wide  range  of  capabilities.  In  earlier  work  on  this  project  we  used  a 
simulator  based  on  full-scale  operational  equipment  to  develop  a  mariner  assessment  procedure 
for  ARPA  operation.  However,  there  are  a  number  of  PC-based  ARPA  simulators  providing 
many  of  the  features  and  functions  available  on  the  more  costly  full-scale  simulators.  The 
simulator  evaluation  procedure  we  developed  in  the  present  effort  was  applied  to  two  PC-based 
ARPA  simulators. 

Basis  for  Simulator  Evaluation 

The  present  approach  evaluates  simulators  on  the  basis  of  their  ability  to  support  performance- 
based  mariner  assessments.  These  simulator  evaluation  requirements  are  developed  from  a  set  of 
mariner  assessment  objectives,  which  define  how  a  mariner  must  demonstrate  his  or  her 
proficiency  in  a  given  operational  area.  For  example,  in  the  case  of  ARPA  operation,  one 
mariner  assessment  objective  is  Demonstrate  use  and  limitations  of  ARPA  operational  warnings 
(McCallum,  et  al.,  2000).  Based  on  assessment  objectives  such  as  this  one,  all  further  mariner 
assessment  requirements  (i.e.,  assessment  conditions,  performance  measures,  and  performance 
standards)  are  developed. 

The  assessment  objectives  guiding  mariner  assessment  development  serve  as  the  basis  for 
developing  simulator  evaluation  requirements.  For  example,  to  ensure  thorough  mariner 
assessment,  an  ARPA  simulator  must  be  able  to  replicate  the  operational  warnings  typically 
found  on  an  ARPA.  Because  mariner  assessment  objectives  are  so  fundamental  to  the  present 
approach  to  simulator  evaluation,  at  any  point  it  should  be  possible  to  trace  a  given  simulator 
evaluation  requirement  back  to  the  mariner  assessment  objective  on  which  it  is  based.  We 
designed  the  simulator  evaluation  procedure  in  the  present  report  to  meet  this  requirement. 

Report  Organization 

The  main  body  of  this  report  has  three  sections.  The  first  section,  A  Method  for  Evaluating 
Simulators  Used  in  Performance-based  Assessment,  describes  a  procedure  for  evaluating 
simulators  used  in  mariner  assessment  and  illustrates  how  this  procedure  can  be  applied.  We 
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used  ARPA  simulators  as  an  example  throughout  the  discussion.  The  second  section,  Illustrative 
Analysis  of  ARPA  Simulator  Capabilities,  provides  examples  of  analyses  possible  with  the 
resulting  findings.  Finally  the  Conclusions  and  Recommendations  section  presents  conclusions 
regarding  the  technical  value  and  practical  applications  of  this  method,  as  well  as 
recommendations  for  refining  and  implementing  the  general  evaluation  method  and  our  ARPA 
protocol. 

In  addition  to  the  main  body,  three  appendices  provide  the  analytical  basis  for  the  evaluation  and 
the  method  for  evaluating  ARPA  simulators.  Appendix  A,  ARPA  Simulator  Evaluation 
Objectives,  Evaluation  Conditions,  and  Evaluation  Criteria,  documents  the  requirements  for  the 
ARPA  simulator  evaluation.  Appendix  B,  ARPA  Simulator  Evaluation  Protocol,  provides  the 
evaluation  protocol  that  we  developed  and  applied  in  the  evaluation  of  two  ARPA  simulators. 
Appendix  C  contains  a  set  of  worksheets  for  compiling  and  analyzing  evaluation  results. 


3 


A  METHOD  FOR  EVALUATING  SIMULATORS  USED  IN 
PERFORMANCE-BASED  ASSESSMENT 


Figure  1  depicts  the  present  method  for  developing  and  conducting  a  simulator  evaluation.  It  is  a 
five-step  process.  In  the  first  step,  performance-based  mariner  assessment  requirements  are 
defined.  The  performance-based  assessment  requirements  are  based  on  an  analysis  of  the 
mariner  assessment  objectives.  These  assessment  objectives  are  identified  through  a  review  of 
mariner  skill  and  knowledge  requirements  in  specified  operational  areas.  Basic  mariner 
assessment  requirements  include  assessment  conditions,  performance  measures,  and  performance 
standards.  In  the  second  step,  simulator  evaluation  objectives  (i.e.,  the  specific  items  on  which 
the  simulator  is  evaluated),  simulator  evaluation  conditions,  and  simulator  evaluation  criteria  are 
defined.  All  three  are  derived  from  the  performance-based  assessment  requirements  defined  in 
the  first  step. 

In  the  third  step,  the  evaluation  protocol  is  developed.  An  evaluation  protocol  is  a  plan  for 
executing  a  simulator  evaluation.  The  protocol  is  organized  by  simulator  evaluation  objective, 
and  is  divided  into  four  categories:  exercise  programming,  equipment  set-up,  simulation,  and 
debriefing.  In  the  fourth  step,  the  simulator  evaluation  is  conducted.  A  separate  evaluation 
should  be  conducted  by  at  least  two  evaluators  to  ensure  the  results  of  the  evaluation  are  reliable. 
Finally,  in  the  fifth  step,  the  findings  of  the  evaluation  are  summarized  and  analyzed.  This 
section  contains  a  discussion  of  these  five  steps,  along  with  examples  of  how  we  applied  this 
method  to  the  evaluation  of  two  ARPA  simulators. 

For  the  ARPA  simulator  evaluation,  we  evaluated  two  PC-based  simulators.  We  selected 
simulators  with  different  processing  characteristics  and  cost  to  ensure  our  evaluation  protocol 
was  sufficiently  flexible  for  application  to  a  range  of  simulators,  and  sufficiently  sensitive  to 
discriminate  among  them.  Additionally,  because  simulator  manufacturers  frequently  change  and 
upgrade  simulator  capabilities  and  offer  systems  in  multiple  configurations,  we  opted  not  to 
identify  the  two  simulators  by  name.  Instead,  we  refer  to  them  as  Simulator  X  and  Simulator  Y. 

In  general.  Simulator  X  was  a  lower-cost  system  with  fewer  simulator  features.  Simulator  Y  was 
a  higher  cost  system  with  a  wider  range  of  features  and  functions.  Simulator  X  was  designed  to 
mimic  the  display  information  of  an  ARPA  unit,  but  not  to  replicate  actual  ARPA  processing 
characteristics.  Simulator  Y  was  designed  to  duplicate  the  processing  characteristics  of  an  actual 
ARPA.  During  the  evaluations,  we  had  the  cooperation  and  participation  of  both  simulator 
manufacturers.  Their  participation  greatly  assisted  us  in  understanding  and  considering  the 
capabilities  of  their  simulators. 
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Figure  1.  Method  for  evaluating  simulators. 


Step  1:  Define  Performance-Based  Mariner  Assessment  Requirements 

The  first  step  in  developing  a  simulator  evaluation  protocol  is  to  define  the  requirements  of  the 
performance-based  mariner  proficiency  assessment  that  will  be  conducted  using  the  simulator. 
The  mariner  assessment  requirements  include  the  assessment  objectives,  assessment  conditions, 
performance  measures  and  performance  standards.  This  step  can  be  time  consuming  but 
simulator  evaluation  is  not  its  primary  purpose.  These  requirements  form  the  basis  of  mariner 
assessments  and  need  to  be  developed,  whether  a  simulator  will  be  used  or  whether  assessment 
will  take  place  in  another  setting.  For  a  detailed  discussion  of  these  concepts,  see  McCallum  et 
al.  (2000).  For  helpful  materials,  including  a  manual,  on  how  to  develop  assessments,  see 
McCallum,  Forsythe,  Barnes,  Smith,  Macaulay,  Sandberg,  Murphy,  &  Jackson  (2000). 

The  mariner  assessment  objectives  are  the  critical  requirements  of  job  performance  that  can  be 
measured  and  assessed.  These  objectives  should  reflect  the  skills  and  knowledge  required  for  a 
job  in  a  specified  operational  area.  All  of  the  corresponding  mariner  assessment  requirements  the 
mariner  assessment  conditions,  performance  measures,  and  performance  standards  should  be 
based  on  the  assessment  objectives.  Ultimately,  all  of  the  mariner  assessment  requirements 
should  be  based  on  a  review  of  job  and  task  requirements  in  the  operational  setting.  Resources 
available  to  support  this  review  include  the  STCW  Code  (IMO,  1996),  the  U.S.  Code  of  Federal 
Regulations,  documented  job  procedures,  technical  manuals,  and  knowledgeable  job  incumbents. 

The  growing  trend  toward  performance-based  assessment  in  the  U.S.  maritime  industry  may 
mean  that  in  the  future,  developers  could  leverage  documented  mariner  assessment  procedures 
for  use  in  simulator  evaluation  protocol  development.  However,  at  the  present  time  few 
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documented  performance-based  mariner  assessment  procedures  exist  in  the  public  domain.  One 
that  is  available  is  the  ARPA  assessment  developed  for  McCallum  et  al.  (2000).  This  procedure 
is  our  source  for  the  ARPA  assessment  requirements  used  in  the  present  simulator  evaluation 
protocol. 

McCallum  et  al.  (2000)  specified  27  performance-based  ARPA  assessment  objectives.  The 
objectives  requiring  use  of  a  simulator  are  listed  in  Appendix  A  in  the  column  labeled  “Mariner 
Assessment  Objective.”  These  assessment  objectives  were  organized  under  six  operational 
categories  that  have  general  applicability  to  assessment  objectives  for  the  operation  of  complex 
equipment  (see  Table  1).  It  is  reasonable  to  expect  that  a  set  of  assessment  objectives  addressing 
the  operation  of  various  types  of  shipboard  equipment,  such  as  an  ECDIS  or  a  GMDSS,  could  be 
organized  under  the  general  headings  presented  in  Table  1 .  As  an  example,  one  of  the  ARPA 
mariner  assessment  objectives  is  1.2,  Selection,  as  appropriate,  of  required  speed  and  compass 
input  to  ARPA,  which  was  in  the  category  corresponding  to  “equipment  initialization.” 

Table  1.  Assessment  objective  categories. 

(1)  Equipment  initialization. 

(2)  Basic  understanding  of  equipment  output. 

(3)  Technical  limitations  of  the  equipment. 

(4)  Advanced  technical  operations. 

(5)  Broad  application  of  skill  and  knowledge  to  the  job. 

(6)  Operational  warnings  and  systems  tests. 


After  the  performance-based  assessment  objectives  are  specified,  the  conditions  for  assessing 
mariner  performance  corresponding  to  each  objective  should  be  identified.  The  assessment 
conditions  define  the  setting,  tools,  reference  aids  and  safety  precautions  that  are  required  for 
assessment  of  mariner  proficiency.  The  conditions  for  each  objective  should  be  precisely 
specified  so  that  comparable  conditions  can  be  replicated  from  one  assessment  session  to  the 
next.  Assessment  conditions  can  vary  widely  in  their  detail  and  complexity,  as  governed  by  the 
technical  content  of  a  given  mariner  assessment  objective.  For  instance,  some  objectives  may 
require  conditions  in  which  the  equipment  is  simply  initialized  (e.g.,  to  demonstrate  basic 
operational  set-up  features  of  the  equipment),  whereas  other  objectives  may  require  that  a 
specific  simulated  exercise  be  running  (e.g.,  to  demonstrate  technical  limitations  of  the 
equipment). 

For  example,  one  of  the  ARPA  simulator  evaluation  objectives  specified  in  McCallum  et  al. 
(2000)  requiring  the  use  of  a  simulator  is  2.5,  Selection  of  vector  time  scale  (see  Appendix  A). 
We  tested  this  objective  during  the  equipment  set-up  phase  before  an  ARPA  test  exercise.  No 
exercise  needed  to  be  running  for  the  assessment  of  this  particular  objective.  However,  mariner 
assessment  objective  2.2  Appreciation  of  the  uses,  benefits,  and  limitations  of  ARPA  operational 
warnings,  required  a  specific  exercise  to  be  running  to  test  the  mariner’s  response  to  operational 
warnings. 

The  other  basic  requirements  for  a  performance-based  assessment  are  the  specification  of 
performance  measures  and  standards.  A  performance  measure  is  a  recordable,  observable  action, 
or  indication  of  an  action.  A  performance  standard  is  an  established  minimum  level  of 
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performance  based  on  relevant  assessment  criteria.  Taken  together,  the  assessment  measures  and 
standards  determine  what  specific  mariner  performance  must  be  elicited  and  recorded  during 
assessment. 

In  their  ARPA  assessment,  McCallum  et  al.  (2000)  specified  three  performance  measures  for 
manner  assessment  objective  2.2,  Appreciation  of  the  uses,  benefits,  and  limitations  of  ARPA 
operational  warnings.  .  Table  2  shows  these  performance  measures  and  their  respective 
performance  standards.  The  candidate  was  measured  on  his  or  her  performance  in  setting  safe 
limits,  and  in  responding  to  safe  limit  warnings  and  guard  zone  warnings.  To  meet  the 
performance  standards,  the  candidate  had  to  set  safe  limits  in  accordance  with  the  assessor’s 
instructions,  and  he  or  she  had  to  identify  the  safe  limit  warning  and  the  guard  zone  warning 
correctly. 


Table  2.  Performance  measures  and  standards  for  ARPA 


mariner  assessment  objective  2,2. 


Mariner  Assessment  Objective 

Performance  Measure 

Performance  Standard 

2.2  Appreciation  of  the  uses, 
benefits,  and  limitations  of 
ARPA  operational  warnings 

2.2.1  Safe  limit  setting 

Safe  limit  set  in  accordance  with 
assessor  instructions 

2.2.2  Safe  limit  warning 

Safe  limit  warning  correctly 
identified 

2.2.3  Guard  zone  warning 

Guard  zone  warning  correctly 
identified 

Step  2:  Define  Simulator  Evaluation  Objectives ,  Evaluation 
Conditions,  and  Evaluation  Criteria 


In  this  step,  the  simulator  evaluation  objectives,  evaluation  conditions,  and  evaluation  criteria  are 
defined.  These  simulator  evaluation  requirements  should  be  derived  from  the  mariner 
assessment  requirements  developed  in  the  first  step.  Precise  definitions  of  the  simulator 
evaluation  requirements  are  essential  to  designing  an  evaluation  protocol  that  is  comprehensive 
and  focused  on  the  simulator  s  ability  to  support  mariner  assessment. 

Define  Simulator  Evaluation  Objectives 

The  factors  on  which  the  simulator  is  evaluated  are  called  simulator  evaluation  objectives. 

These  objectives  are  based  on  the  mariner  assessment  objectives  defined  in  the  previous  step,  as 
well  as  on  the  performance  standards  for  simulators  specified  by  the  IMO  in  the  amended  STCW 
Code ,  Section  A-P12,  Part  I,  2. 1-2.6  (IMO,  1996).  Table  3  summarizes  these  general  standards. 


7 


Table  3.  IMO  general  performance  standards  for  simulators. 


(1 )  Ability  to  satisfy  the  specified  assessment  objectives. 

(2)  Ability  to  simulate  the  operating  capabilities  of  the  shipboard  equipment  ^ncernect  to  a  level 
V  '  of  realism  appropriate  to  the  assessment  objectives,  and  include  the  capabilities,  technical 

limitations  and  possible  errors  of  such  equipment. 

(3)  Ability  to  provide  sufficient  behavioral  realism  to  allow  an  assessment  candidate  to  exhibit  the 
knowledge  and  skills  appropriate  to  the  assessment  objectives. 

(4)  Ability  to  provide  an  interface  through  which  a  mariner  can  interact  with  the  equipment  and 
simulated  environment. 

Abilitv  to  Drovide  a  controlled  operating  environment  capable  of  generating  a  variety  of 
'  ’  c“SioXSd^  emergency,  hazardous,  o,  unusual  situations  relevant  to  the  assessment 

objectives. 

(6)  Ability  to  permit  an  assessor  to  control,  monitor,  and  record  exercises  for  the  effective 

assessment  of  candidate  performance. _ _ _ _ _ _. — - - - 


The  first  three  standards  relate  to  the  simulator’s  ability  to  replicate,  in  a  realistic  manner,  the 
operational  capabilities  and  limitations  of  the  shipboard  equipment.  Operational  capabilities  and 
limitations  refer  to  the  functions  and  features  of  the  equipment  relative  to  its  intended  use.  or 
example,  one  ARPA  function  is  to  acquire  and  track  vessel  targets.  This  function  has  some 
limitations,  such  as  delays  in  computing  the  speed  and  heading  of  multiple  targets.  Simulators 
should  be  evaluated  on  their  ability  to  replicate  operational  limitations  such  as  this  one. 

The  fourth  standard  relates  to  the  simulator’s  ability  to  provide  a  display  interface  through  which 
the  mariner  can  view  the  operations  of  the  simulator  and  interact  with  the  equipment  and 
simulated  environment.  The  fifth  standard  relates  to  the  simulator’s  ability  to  provide  a 
controlled  operating  environment  capable  of  generating  a  variety  of  conditions  for  use  in 
assessment  exercises.  Finally,  the  sixth  standard  addresses  the  simulator’s  ability  to  provide 
adequate  means  for  an  assessor  to  debrief  a  mariner  following  an  assessment. 

Our  development  of  the  simulator  evaluation  method  was  intended  to  follow  STCW  direction. 

As  summarized  in  Table  3,  the  STCW  Code  requires  that  simulators  used  in  mariner  proficiency 
assessment  possess  an  “interface  through  which  a  mariner  can  interact  with  the  equipment 
(IMO,  1996).  No  mention  is  made,  however,  of  the  necessity  for  simulator  contro  s  to  rep  ica  e 
the  controls  of  a  particular  equipment  manufacturer.  The  evaluation  of  specific  controls  is  a 
possibility  not  only  for  an  ARPA  simulator,  but  also  for  a  PC-based  simulator  of  any  type  of 
equipment.  If  an  evaluation  of  a  simulator’s  capability  to  mimic  specific  controls  is  desired,  such 
objectives  could  be  built  into  the  evaluation  protocol. 

The  six  standards  presented  in  Table  3  can  be  divided  into  four  main  categories  of  simulator 
capability: 

•  Exercise  programming  (standard  5). 

•  Equipment  set-up  (standard  4). 

•  Simulation  (standards  1-3). 

•  Debriefing  (standard  6). 
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The  order  used  to  list  these  four  main  categories  of  simulator  capability  differs  from  the  order 
adopted  by  IMO.  The  current  order  better  reflects  the  sequence  of  steps  one  follows  when 
conducting  a  simulator  evaluation. 

Simulator  evaluation  objectives  should  be  defined  for  each  of  these  categories  of  simulator 
capability.  Exercise  programming  objectives  should  address  the  simulator’s  ability  to  permit  an 
assessor  to  control  exercise  conditions.  For  ARPA  simulators,  this  includes  the  programming  of 
own  ship  hydrodynamic  and  maneuvering  characteristics,  land  masses,  and  environmental 
characteristics  such  as  wind,  current,  etc.  Equipment  set-up  objectives  should  address  all  items 
pertinent  to  initiating  and  setting  up  the  display  and  equipment.  (The  controls  characteristic  of  a 
specific  manufacturer  could  be  included  in  this  category  of  objectives.)  Simulation  objectives 
should  address  all  operational  capabilities  and  technical  limitations  requiring  a  dynamic, 
simulated  exercise,  including  equipment  failures,  errors,  and  alarms.  Lastly,  debriefing 
objectives  should  address  the  simulator’s  ability  to  permit  the  assessor  to  monitor,  record,  replay, 
and  print  the  results  of  exercises. 

As  noted  earlier,  in  addition  to  the  above  standards,  the  simulator  evaluation  objectives  are  based 
on  the  mariner  assessment  objectives  defined  in  the  previous  step.  The  operational  capabilities 
and  limitations  required  for  each  assessment  objective  should  be  identified  and  represented  in  the 
simulator  evaluation  objectives.  As  an  example,  for  our  ARPA  simulator  evaluation,  we  derived 
33  simulator  evaluation  objectives  from  the  27  ARPA  assessment  objectives  specified  in 
McCallum  et  al.  (2000).  These  ARPA  simulator  evaluation  objectives  include  all  of  the 
operational  capabilities,  controls,  and  displays  required  for  a  simulator  to  run  the  ARPA 
assessment  in  McCallum  et  al.  (2000).  An  example  is  mariner  assessment  objective  5.3,  The 
operation  of  the  trial  maneuver  facility.  To  address  this  requirement,  we  defined  a  corresponding 
simulator  evaluation  objective  3.14,  Operation  of  the  trial  maneuver  facility .  Appendix  A 
contains  a  set  of  tables  delineating  all  of  the  evaluation  objectives  we  specified  for  ARPA 
simulators. 

Define  Simulator  Evaluation  Conditions 

After  the  simulator  evaluation  objectives  have  been  determined,  the  evaluation  conditions  for 
each  objective  can  be  defined.  Evaluation  conditions  refer  to  the  context  of  the  equipment’s 
intended  use.  The  context  of  use  includes  the  tasks  that  mariners  perform,  as  well  as  the 
circumstances  influencing  the  behavior  of  the  equipment.  This  context  also  includes 
environmental  conditions  that  could  degrade  the  performance  of  the  equipment.  For  example,  sea 
clutter  is  an  important  evaluation  condition  for  ARPA  simulators,  because  sea  clutter  can  induce 
operator  errors  by  masking  target  information.  An  ARPA  simulator  should  be  evaluated  on  both 
its  ability  to  display  sea  clutter  and  its  ability  to  reproduce  the  masking  of  target  information 
when  sea  clutter  is  present. 

When  defining  simulator  evaluation  conditions,  the  mariner  performance-based  assessment 
requirements  specified  in  step  1  should  be  considered.  These  assessment  requirements  dictate 
the  assessment  conditions  to  be  supported  by  the  simulator,  and  form  the  basis  for  the  evaluation 
conditions.  The  evaluation  conditions  should  specify  whether  the  simulator  should  be  evaluated 
in  a  dynamic  mode  (i.e.,  simulating  an  operational  exercise),  or  in  a  static  mode  (i.e.,  initialized 
but  not  running  an  exercise).  They  should  also  specify  the  situation  to  replicate  (e.g.,  location 
and  vessels  involved)  and  actions  to  perform  (e.g.,  appropriate  course  change). 
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Evaluation  conditions  can  take  the  form  of  standardized  exercises.  The  standardized  exercises 
used  for  simulator  evaluation  can  be  the  ones  developed  for  the  mariner  assessment,  or  they  can 
be  more  simple  exercises  designed  to  test  specific  simulator  operations.  Ideally,  these  exercises 
should  be  developed  with  the  assistance  of  individuals  knowledgeable  about  the  operational 
capabilities  of  the  actual  shipboard  equipment. 

For  our  ARPA  simulator  evaluation,  we  used  standardized  exercises  developed  for  McCallum  et 
al.  (2000).  Within  each  exercise,  we  specified  the  evaluation  conditions  for  each  evaluation 
objective,  and  then  categorized  the  conditions  as  either  dynamic  or  static.  Simulator  evaluation 
objective  3.5,  Use  of  graphic  representation  of  danger  areas ,  is  an  example  of  an  objective  that 
we  evaluated  using  a  dynamic  exercise,  because  it  was  necessary  to  observe  the  navigation  of 
own  ship  close  to  danger  areas  to  see  how  the  simulator  represented  danger  areas.  Simulator 
evaluation  objective  2.5,  Selection  of  vector  time  scale,  is  an  example  of  an  objective  that  we 
evaluated  when  the  simulator  was  in  a  static  mode.  To  meet  this  objective,  an  ARPA  simulator 
had  to  provide  both  adjustable  time  and  fixed  time  scales,  and  it  had  to  indicate  which  vector 
time  scale  was  in  use.  No  specific  exercise  needed  to  be  running  to  evaluate  these  capabilities. 
Appendix  A  lists  the  conditions  for  each  evaluation  objective  that  we  specified  for  ARPA 
simulators. 

Define  Simulator  Evaluation  Criteria 

When  the  simulator  evaluation  objectives  and  evaluation  conditions  have  been  defined,  the 
evaluation  criteria  can  be  specified.  Evaluation  criteria  refer  to  the  simulator’s  ability  to  provide 
the  specific  feature  (a  control  or  a  display)  needed  to  meet  a  specific  evaluation  objective.  A 
control  enables  the  operator  to  perform  a  given  function  (e.g.,  a  rotary  knob  used  to  select  a 
heading).  A  control  criterion  can  also  represent  the  underlying  function  enabled  by  a  control.  A 
display  is  a  visual  or  auditory  representation  of  the  function  or  the  environment  (e.g.,  North-up 
display  of  a  vessel’s  course). 

Simulator  evaluation  criteria  can  be  based  on  one  or  more  of  the  following:  STCW  simulator 
requirements  (IMO,  1996);  IMO  performance  standards  for  the  actual  equipment  (IMO,  1971, 
1979);  and  the  requirements  needed  to  satisfy  the  specified  mariner  assessment  objectives  (Bole 
&  Dineley,  1990;  McCallum  et  al.,  2000).  For  example,  simulator  evaluation  objective  2.5, 
Selection  of  vector  time  scale,  has  evaluation  criteria  that  are  based  on  both  STCW  simulator 
requirements  and  IMO  performance  standards  for  actual  equipment.  The  evaluation  criteria  for 
this  objective  consist  of  one  control  feature,  2. 5. Cl,  Ability  to  select  adjustable  time  scale  or 
fixed  time  scale ;  and  one  display  feature,  2.5  .Dl,  Indication  of  time  scale  of  vector  in  use.  The 
tables  in  Appendix  A  contain  the  complete  list  of  control  and  display  criteria  we  specified  for  the 
ARPA  simulator  evaluation  objectives. 

Step  3:  Develop  Simulator  Evaluation  Protocol 

The  simulator  evaluation  objectives,  evaluation  conditions,  and  evaluation  criteria  that  were 
defined  in  the  previous  step  provide  the  foundation  for  the  simulator  evaluation  protocol.  The 
protocol  can  be  organized  around  the  individual  simulator  evaluation  objectives,  and  divided  into 
four  sections  corresponding  to  the  simulator  evaluation  objective  categories:  exercise 
programming,  equipment  set-up,  simulation,  and  debriefing.  The  protocol  should  address  the 
following  information  for  each  objective: 
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•  Evaluation  conditions. 

•  Evaluation  criteria  for  controls. 

•  Evaluation  criteria  for  displays. 

•  Availability  of  specific  control  or  display. 

•  Simulator  performance  rating  for  each  control  and  display  criterion. 

•  Comments. 

•  Specifications  worksheet. 

The  evaluation  conditions  tell  the  evaluators  what  specific  conditions  need  to  be  present  to 
evaluate  each  objective.  Evaluators  need  this  information  to  ensure  comparable  conditions  exist 
for  each  evaluation. 

The  evaluation  criteria  for  controls  specify  which  control  features  should  be  evaluated.  Our 
evaluation  criteria  address  the  underlying  processing  characteristics  and  limitations  represented 
by  a  simulator’s  controls,  rather  than  strictly  the  physical  resemblance  of  the  controls  to  actual 
equipment.  This  follows  the  approach  set  forth  in  the  STCW  standards  for  simulators  (see  Table 
3).  However,  the  physical  fidelity  of  controls  may  be  a  consideration  in  some  applications  or 
assessments  where  actual  equipment  operation  is  an  objective. 

The  evaluation  criteria  for  displays  specify  which  display  features  should  be  evaluated.  All 
controls  and  displays  relating  to  the  evaluation  objectives  should  be  addressed,  and  there  may  be 
more  than  one  control  and  display  for  each  objective. 

The  last  three  areas  of  the  protocol — availability  of  specific  control  or  display,  simulator 
performance  rating  for  each  control  and  display  criterion,  and  comments —  are  for  recording 
evaluator  observations.  An  evaluator  can  note  whether  or  not  the  specified  control  or  display  was 
available  on  the  simulator,  and  he  or  she  can  rate  how  well  the  simulator  satisfied  the  evaluation 
criteria.  A  variety  of  different  rating  scales  can  be  used.  At  a  minimum,  the  rating  scale  should 
permit  evaluators  to  make  a  distinction  among  simulators  that  meet,  partially  meet,  and  do  not 
meet,  the  evaluation  criteria.  By  providing  comments,  an  evaluator  can  note  any  other  pertinent 
information  that  should  be  included  in  the  evaluation. 

In  addition,  the  evaluation  protocol  should  include  a  specifications  worksheet  for  each  simulator 
evaluated.  Equipment  specifications  are  useful  for  describing  and  comparing  simulators.  Items 
to  include  on  the  specifications  worksheet  are  the  manufacturer,  model,  hardware,  software, 
network  configuration,  actual  equipment  interface  of  the  simulator  (i.e.,  the  manufacturer  and 
model  that  the  simulator  replicates,  if  any),  cost,  etc.  The  list  of  items  on  this  worksheet  depends 
on  the  type  of  simulator  being  evaluated. 

In  developing  the  protocol,  instructions  should  be  written  for  evaluators  so  they  have  a 
standardized  process  for  rating  each  simulator.  Written  instructions  should  define  the  rating 
scales  and  rating  criteria,  as  well  as  the  purpose  of  each  section  of  the  protocol.  Explicit 
instructions  can  help  to  ensure  evaluation  results  are  comparable  across  simulators. 

The  protocol  we  used  for  the  ARPA  simulator  evaluation  is  provided  in  Appendix  B.  This 
protocol  includes  a  specifications  worksheet  and  a  set  of  standardized  exercises  designed  to 
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replicate  operational  limitations  of  actual  shipboard  equipment.  It  also  includes  a  set  of  exercises 
designed  to  evaluate  the  conditions  and  criteria  necessary  to  support  mariner  assessment. 

Step  4:  Conduct  Simulator  Evaluation 

The  fourth  step  in  evaluating  simulators  is  to  conduct  the  evaluation  to  determine  how  well  the 
simulator  supports  performance-based  assessment.  When  conducting  an  evaluation,  it  is 
beneficial  to  have  the  cooperation  and  participation  of  the  simulator  manufacturers,  to  assist 
evaluators  in  programming  exercises  and  operating  the  equipment,  as  well  as  to  ensure 
understanding  and  consideration  of  all  the  simulator’s  capabilities.  In  addition,  manufacturers 
may  benefit  from  a  thorough  evaluation  by  discovering  the  strengths  and  weaknesses  of  their 
products. 

If  possible,  two  or  more  evaluators  should  participate  in  the  evaluation.  Comparing  and 
integrating  the  findings  of  multiple  raters  result  in  a  more  reliable  evaluation.  Also,  an  objective 
evaluation  requires  evaluators  to  follow  well-defined  procedures  for  administering  the  protocol. 
The  procedures  might  vary  depending  on  the  nature  of  the  simulator  being  evaluated.  For 
example,  the  evaluation  of  a  single  component  simulator,  such  as  an  ECDIS,  would  be  much  less 
complex  than  the  evaluation  of  a  multi-component  simulator,  such  as  an  engine  room  simulator. 
Table  4  summarizes  the  procedures  we  followed  when  conducting  evaluations  of  the  two  ARPA 
simulators. 


Table  4.  Procedures  for  conducting  a  simulator  evaluation. 


(1)  Provide  the  manufacturer  with  the  evaluation  conditions  (including  standardized  exercises)  to  be 
programmed  ahead  of  time,  if  applicable. 

(2)  Ensure  a  knowledgeable  person  is  available  to  demonstrate  the  simulator’s  capabilities. 

(3)  Ask  the  manufacturer  to  provide  the  specifications  for  the  equipment  being  evaluated. 

(4)  Evaluate  each  item  in  the  order  presented:  programming  capabilities,  set-up  capabilities, 
simulation  capabilities,  and  debriefing  capabilities.  Ensure  all  simulator  evaluation  objectives 
have  been  evaluated. 

(5)  Evaluate  each  item  requiring  a  dynamic  evaluation  using  the  appropriate  exercise.  Run  each 
exercise  separately. 

(6)  Conclude  the  evaluation  process  by  asking  the  manufacturer  to  discuss  any  features  that  might 
have  been  overlooked  during  the  evaluation. 


Before  evaluating  each  ARPA  simulator,  we  forwarded  an  overview  of  the  evaluation  objectives 
and  a  detailed  description  of  the  evaluation  exercise  scenarios  to  each  manufacturer. 
Subsequently,  we  sent  a  team  of  three  evaluators  to  conduct  the  simulator  evaluations  at  each 
company.  During  these  evaluations,  evaluators  observed  the  simulators  running  the  standardized 
exercises,  and  each  evaluator  independently  completed  an  evaluation  form.  Evaluators  used  a 
rating  scale  of  yes,  partial,  or  no.  A  yes  score  indicated  the  simulator  met  the  criterion.  A 
partial  score  indicated  the  simulator  partially  met  the  criterion,  and  a  no  score  indicated  the 
simulator  did  not  meet  the  criterion. 
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Step  5:  Summarize  and  Analyze  Findings 

The  final  step  in  a  simulator  evaluation  is  to  summarize  and  analyze  the  findings.  Summarizing 
the  findings  entails  integrating  observations  and  scores  across  evaluators  for  each  evaluation 
criterion.  Analyzing  the  findings  entails  combining  and  extracting  selected  simulator  evaluation 
criterion  scores  to  address  specific  issues  with  respect  to  the  capability  of  a  simulator  to  support 
mariner  assessment. 

To  summarize  the  results  of  the  evaluation,  evaluators’  ratings  and  observations  should  first  be 
combined  for  each  simulator  evaluation  criterion.  Methods  for  combining  differing  observations 
can  be  based  on  either  a  consensus-building  approach  or  an  averaging  approach.  Using  a 
consensus-building  approach,  differing  observations  and  scores  are  identified  and  discussed 
among  the  evaluators  until  an  agreement  concerning  the  observation  or  score  is  reached.  This 
approach  is  time-consuming  but  has  the  advantage  of  addressing  subtle  or  highly  technical  issues 
reflecting  a  simulator’s  capabilities.  Using  an  averaging  approach,  observations  and  ratings  are 
combined  by  determining  the  central  tendency  among  evaluators.  This  approach  is  more 
efficient  but  may  tend  to  obscure  subtle  or  highly  technical  concerns  regarding  simulator 
characteristics. 

Using  the  resulting  simulator  evaluation  criteria  scores  as  a  basis,  the  evaluation  team  can 
conduct  a  series  of  analyses  addressing  the  capabilities  of  the  simulator  or  simulators  under 
evaluation.  Three  general  types  of  analysis  can  be  conducted.  First,  the  scores  for  each  simulator 
evaluation  criterion  can  be  combined  to  obtain  scores  for  each  of  the  separate  simulator 
evaluation  objectives.  Second,  the  separate  evaluation  objective  scores  can  be  combined  to 
evaluate  capabilities  with  respect  to  each  of  the  four  evaluation  categories:  exercise 
programming,  equipment  set-up,  simulation,  and  debriefing.  Third,  selected  simulator  evaluation 
scores  can  be  extracted  and  summarized  to  address  specific  issues,  such  as  the  capability  to 
support  individual  mariner  assessment  objectives  and  IMO  simulator  requirements.  These  types 
of  analyses  are  illustrated  in  the  next  section  by  the  results  of  our  ARPA  evaluation. 
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ILLUSTRATIVE  ANALYSIS  OF  ARPA  SIMULATOR  CAPABILITIES 


Using  our  evaluators’  observations  and  scores  for  the  two  ARPA  simulators,  we  performed  the 
three  types  of  analyses  listed  in  the  previous  section.  Selected  results  of  our  analyses  are 
presented  below  to  illustrate  both  the  analysis  procedures  and  the  types  of  comparisons  that  can 
be  made.  The  worksheets  that  we  used  to  perform  these  analyses  are  presented  in  Appendix  C. 

Capability  of  simulators  to  support  individual  simulator  evaluation  objectives.  In  the 

present  ARPA  simulator  evaluation,  scores  for  the  three  separate  evaluators  were  reviewed, 
discrepancies  among  evaluators  were  identified,  and  each  issue  was  discussed  until  a  consensus 
score  for  each  evaluation  criterion  was  obtained.  By  assigning  a  numerical  value  of  1  {yes),  0.5 
{partial),  or  0  {no),  we  calculated  simulator  scores  for  each  evaluation  objective.  These  scores 
allowed  us  to  identify  the  relative  strengths  and  weaknesses  of  each  simulator. 

Table  5  compares  the  scores  for  Simulator  X  and  Simulator  Y  on  each  of  the  evaluation  criteria 
corresponding  to  simulator  evaluation  objective  2. 1,  Selection  of  display  presentation, 
orientation,  and  vector  mode.  This  table  indicates  that  Simulator  X  fully  met  six  criteria, 
partially  met  three  criteria,  and  did  not  meet  two  criteria,  resulting  in  a  score  of  7.5  for  simulator 
evaluation  objective  2. 1 .  In  comparison.  Simulator  Y  fully  met  nine  of  the  criteria  and  did  not 
meet  two  of  the  criteria,  resulting  in  a  score  of  9.0  for  this  simulator  evaluation  objective. 

Table  5.  Summary  of  simulator  capabilities  for  evaluation  objective  2.1,  selection  of  display 
presentation,  orientation,  and  vector  mode. 


Simulator  Evaluation  Criterion  (C  =  Control,  D  =  Display) 

Evaluation  Criterion  Met 

Simulator  X  Simulator  Y 

2.1. Cl 

Ability  to  toggle  between  sea-  and  ground-stabilized 
modes 

No 

No 

2.1.D1 

Indication  of  display  mode 

No 

No 

2.1. C2 

Ability  to  toggle  between  North-up,  and  either  course-up 
or  head-up  azimuth  stabilization 

Partial 

Yes 

2.1. D2 

Indication  of  display  orientation  mode 

Yes 

Yes 

2.1. C3 

Ability  to  toggle  between  relative  and  true  motion 

Yes 

Yes 

2.1. D3 

Indication  of  display  vector  mode 

Partial 

Yes 

2.1.C4 

Ability  to  use  ARPA  on  the  following  ranges:  (a)  3  or 

4  miles,  and  (b)  12  or  16  miles 

Yes 

Yes 

2.1.C5 

Fixed  range  rings  available 

Yes 

Yes 

2.1.D5.1 

Indication  of  range  scale  in  use 

Yes 

Yes 

2.1. D5.2  Indication  of  distance  between  range  rings 

Partial 

Yes 

2.1.  C6 

Availability  of  variable  range  marker  (VRM) 

Yes 

Yes 

Summary  Score  (Yes  =  1,  Partial  =  0.5,  No  =  0) 

7.5 

9.0 

The  scoring  approach  illustrated  in  Table  5  represents  a  modest  level  of  technical  sophistication. 
A  more  advanced  approach  would  be  to  apply  different  weights  to  the  separate  simulator 
evaluation  criteria  prior  to  calculating  summary  scores.  Higher  weights  would  indicate  those 
criteria  that  are  considered  relatively  more  important  than  other  criteria.  Scores  would  be 
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multiplied  by  the  respective  weight  for  each  criterion  to  obtain  a  weighted  score.  Valid  criterion 
weights  could  be  obtained  through  structured  reviews  with  subject  matter  experts. 

Capability  of  simulators  to  support  each  of  the  four  general  simulator  evaluation 

categories.  After  analyzing  simulator  capabilities  at  the  detailed  level  of  individual  simulator 
evaluation  criteria,  analyses  can  address  the  more  general  issues  corresponding  to  the  four 
evaluation  categories.  Here,  the  scores  for  the  individual  evaluation  objectives  can  be  summed 
within  the  corresponding  evaluation  category  to  provide  summary  scores  for  each  category. 
These  more  general  scores  provide  a  broader  basis  for  evaluating  the  simulator’s  main  strengths 
and  weaknesses.  Figure  2  depicts  the  percentage  of  possible  criterion  scores  met  by  each  ARPA 
simulator  within  each  simulator  evaluation  category.  This  figure  reveals  consistently  high  scores 
for  Simulator  Y  (between  80  and  98  percent  of  the  criteria  were  met  for  each  evaluation 
objective);  and  more  varied,  but  consistently  lower  scores  for  Simulator  A"  (between  20  and  56 
percent  of  the  criteria  were  met). 


Evaluation  Category 


Figure  2.  Percentage  of  criteria  met  by  each  ARPA  simulator,  in  four  evaluation 
categories. 

Capability  of  simulators  to  support  mariner  assessment.  A  simulator’s  capability  to  support 
the  mariner  assessment  objectives  is  a  fundamental  consideration  in  the  analysis  of  evaluation 
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results.  Analysis  of  these  issues  can  be  conducted  by  compiling  simulator  evaluation  objective 
scores  corresponding  to  each  of  the  mariner  assessment  objectives.  Figure  3  compares  the 
capabilities  of  Simulator  X and  Simulator  Y  to  support  the  six  assessment  objective  categories 
specified  in  McCallum  et  al.  (2000).  This  figure  reveals  consistently  high  percentage  scores  for 
Simulator  Y  (between  83  and  100  percent  of  the  assessment  objectives  were  supported);  and 
more  varied  percentage  scores  for  Simulator  X  (between  17  and  100  percent  of  the  objectives 
were  supported). 
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Figure  3.  Percentage  of  assessment  objectives  supported  by  each  ARPA  simulator,  in  six 
assessment  objective  categories. 

Interpretation  of  the  findings.  Overall,  our  evaluation  summaries  indicated  that  Simulator  X 
had  limited  capabilities  in  each  of  the  four  simulator  evaluation  categories:  exercise 
programming,  equipment  set-up,  simulation,  and  support  for  debriefing.  As  an  example,  the 
system  could  simulate  landmasses  and  environmental  conditions,  but  it  did  not  provide  flexibility 
in  specifying  either  the  strength  or  weakness  of  the  conditions.  The  main  strength  of  its 
simulation  was  its  capability  to  support  parallel  indexing.  It  had  minimal  capability  to  record 
exercises  and  support  debriefing.  On  the  other  hand,  Simulator  Y  supported  the  bulk,  although 
not  all,  of  the  requirements  in  each  of  these  categories.  Simulator  Fs  strengths  included  its 
ability  to  generate  complex  and  varied  exercise  conditions  and  its  ability  to  record  exercises. 
Weaknesses  in  its  simulation  included  an  inability  to  specify  danger  areas  on  one  of  the  radar 
display  interfaces  we  tested. 
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Simulator  X  also  had  limitations  in  supporting  mariner  assessment  objectives.  This  system 
mimicked  ARPA  display  features,  but  not  the  underlying  processing  characteristics.  For 
example,  its  target  display  information  did  not  reflect  the  temporary  processing  delays  and 
inaccuracies  typical  of  actual  ARPA  systems.  For  use  in  mariner  assessment,  Simulator  X  would 
need  augmented  vessel  target  processing  capabilities  so  assessment  candidates  could  experience 
the  conditions  exhibited  by  an  actual  ARPA  unit.  Simulator  Y  offered  greater  support  for  mariner 
assessment.  Its  ability  to  duplicate  the  actual  display,  control,  and  processing  characteristics  of 
an  ARPA  unit  allowed  it  to  generate  varied  exercise  scenarios  and  faithfully  simulate  target  ship 
processing. 

Generalizing  from  the  analysis.  In  this  report,  we  have  presented  and  illustrated  a  very 
structured  method  for  the  evaluation  and  analysis  of  simulators  in  their  capability  to  support  an 
extremely  demanding  application —  performance-based  assessment  of  mariner  proficiency.  For 
our  ARPA  example,  we  have  identified  the  particular  strengths  and  weaknesses  of  two  simulators 
in  their  capabilities  for  mimicking  individual  ARPA  features,  for  providing  broader  ARPA 
simulator  functions,  and  finally  for  supporting  the  IMO  requirements  for  assessment  of  mariner 
proficiency  in  ARPA  operation.  Whereas  both  simulators  support  many  of  the  assessment 
objectives,  one  more  than  the  other,  neither  one  supports  100  percent  of  the  objectives.  Could 
either  one  of  the  simulators  we  evaluated  support  assessment  of  mariner  proficiency  in  ARPA 
operation?  The  answer  is  not  a  simple  “yes”  or  “no.”  The  mariner  assessment  requirements 
cannot  all  be  met,  even  by  the  more  capable  of  the  two  simulators.  However,  the  potential 
advantages  of  using  simulators  rather  than  real  equipment  remain. 

A  more  productive  question  to  be  answered  by  this  structured  evaluation  might  be,  “What  is  the 
best  use  of  proposed  simulator  technology?”  A  simulator  might  be  used  for  a  preliminary 
assessment  of  mariner  performance  to  ensure  that  the  individual  is  ready  to  make  the  best  use  of 
an  opportunity  for  assessment  on  real  equipment  in  a  laboratory  or  at  sea.  As  an  alternative,  a 
preliminary  assessment  on  a  simulator  might  be  augmented  by  a  later,  more  limited  assessment 
on  real  equipment.  Given  either  of  these  approaches,  a  decision  would  have  to  be  made  as  to 
whether  the  greater  effectiveness  of  a  more  costly  simulator  or  of  assessment  on  real  equipment 
is  worth  the  increased  cost.  The  detailed  evaluation  method  that  we  have  presented  is  a  tool  not 
only  for  the  assessment  developer,  but  also  for  the  simulator  designer.  The  evaluation  identifies 
weak  features  and  the  potential  value  of  their  improvement  to  the  user,  especially  for  mariner 
assessment.  After  identifying  the  weaknesses,  the  evaluator  can  consider  the  value  of  a  potential 
improvement  in  relation  to  its  cost  to  the  manufacturer  and  to  future  buyers.  We  have  proposed  a 
systematic  method  for  evaluating  simulators  and  must  leave  it  to  others  in  the  maritime  industry 
to  design  a  broader  program  of  performance-based  assessments  that  benefit  from  the  capabilities 
of  simulators.  A  parallel  effort  to  ours,  to  systemically  identify  the  features  needed  by  engine 
room  simulators  to  support  mariner  assessment,  reached  a  similar  conclusion,  that  simulators 
need  to  be  incorporated  into  a  broader  program  of  mariner  assessment  (Stutman,  1999). 
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CONCLUSIONS  &  RECOMMENDATIONS 


The  following  are  our  conclusions  and  recommendations,  based  on  our  experiences  and  findings 
from  the  present  research  effort.  The  conclusions  address  the  technical  feasibility,  practicality, 
and  potential  applications  of  this  method.  The  recommendations  identify  actions  for  the 
refinement  and  implementation  of  both  the  general  simulator  evaluation  method  and  the  ARPA 
simulator  evaluation  protocol. 

Conclusions 

Based  on  our  experience  and  findings  during  the  development  and  application  of  the  simulator 
evaluation  method,  we  conclude  that  this  approach  is  technically  feasible  and  practical.  In 
addition,  because  this  method  is  based  on  the  requirements  of  mariner  performance-based 
assessment,  we  also  conclude  it  can  be  applied  in  a  broad  range  of  domains.  The  five  major 
conclusions  of  this  report  are  presented  below. 

The  simulator  evaluation  method  is  technically  feasible  and  practical.  We  demonstrated  the 
technical  feasibility  and  practical  value  of  our  approach  for  evaluating  the  capability  of  marine 
simulators  to  support  mariner  assessment.  A  structured  method  for  evaluating  simulators  based 
on  performance-based  assessment  requirements  was  defined.  An  example  evaluation  protocol 
for  ARPA  simulators  was  then  developed  and  successfully  applied  in  the  evaluation  of  two  PC- 
based  ARPA  simulators.  This  application  allowed  us  to  refine  both  the  method  and  evaluation 
procedure.  It  also  allowed  us  to  verify  the  technical  feasibility  and  practical  value  of  the  general 
method  and  the  ARPA  evaluation  protocol. 

The  simulator  evaluation  method  is  fully  compliant  with  STCW  Code  standards  for 
simulators.  The  STCW  Code  establishes  a  set  of  performance  standards  for  simulators 
supporting  mariner  assessments  (see  Table  3).  These  standards  represent  the  basic  requirements 
for  any  simulator  to  be  able  to  support  mariner  assessment.  The  present  method  is  fully 
compliant  with  these  standards.  Each  of  the  STCW  Code  simulator  standards  is  explicitly 
incorporated  in  the  present  method. 

The  simulator  evaluation  method  could  be  applied  to  a  broad  range  of  simulators  in  the 
maritime  and  other  industries.  The  present  application  of  the  simulator  evaluation  method  was 
limited  to  PC-based  ARPA  simulators.  However,  the  method  has  a  much  broader  range  of 
potential  applications.  It  could  be  applied  to  the  full  spectrum  of  ARPA  simulators,  as  well  as  a 
wide  range  of  bridge  and  engine  room  simulators  (e.g.,  ECDIS  simulators,  GMDSS  simulators, 
and  diesel  engine  simulators).  In  addition,  the  method  could  be  applied  to  other  maritime 
simulators  (e.g.,  vessel  loading  simulators  and  vessel  traffic  system  simulators)  or  simulators 
designed  for  assessment  of  performance  in  other  industries  (e.g.,  flight  simulators,  driving 
simulators,  and  power  plant  control  simulators). 

The  simulator  evaluation  method  can  be  generalized  to  the  evaluation  of  training 
simulators.  The  method  described  in  this  report  focused  on  the  evaluation  of  simulators  for  use 
in  mariner  performance  assessment.  Performance  assessment  is  an  important  use  of  simulators, 
but  an  equally  important  use  is  training.  As  in  the  case  of  assessment  procedures,  training 
programs  can  be  developed  with  the  explicit  identification  of  performance  objectives  and 
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performance  measures.  Given  the  specification  of  these  training  requirements,  simulator 
evaluation  criteria  could  be  developed  to  determine  the  capability  of  a  simulator  to  support  a 
training  program. 

Training  institutions,  regulatory  agencies,  and  simulator  manufacturers  can  apply  the 
simulator  evaluation  method.  There  is  a  wide  range  of  potential  users  of  this  evaluation 
method.  The  full  range  of  training  institutions  (academies,  colleges,  and  commercial  training 
centers)  could  use  this  methodology  for  the  selection  of  cost-effective  simulators.  The  USCG  or 
other  regulatory  agencies  also  could  use  the  method  to  develop  standardized  evaluation 
procedures  for  different  types  of  simulators.  In  addition,  simulator  manufacturers  could  use  the 
established  evaluation  standards,  as  well  as  completed  evaluations,  to  determine  those  features 
and  capabilities  that  should  be  modified  in  future  design  upgrades. 

Recommendations 

The  overall  objective  of  this  research  effort  was  to  develop  an  approach  to  simulator  evaluation 
for  use  by  the  USCG  and  the  maritime  industry  in  their  response  to  the  1995  STCW 
amendments.  Given  the  demonstrated  feasibility  and  success  of  this  approach,  we  recommend 
that  the  USCG,  maritime  academies,  simulator  manufacturers,  and  other  organizations  take  the 
following  actions  to  bring  this  approach  into  practice.  The  recommendations  are  organized  into 
two  sections  corresponding  to  the  primary  products  of  this  effort:  the  simulator  evaluation 
method,  and  the  ARPA  evaluation  protocol. 

Simulator  Evaluation  Method 

This  document  provides  a  relatively  general  summary  of  the  simulator  evaluation  method  we 
developed  and  refined.  We  recommend  that  this  description  serve  as  a  guide  for  implementation 
by  the  USCG  and  members  of  the  maritime  educational  community.  We  recommend  the 
following  actions  to  ensure  this  simulator  evaluation  method  best  supports  the  purpose  of 
increasing  the  effectiveness  of  simulator  and  training  course  evaluations  in  the  maritime  industry. 

Distribute  this  simulator  evaluation  protocol.  A  complete  and  general  method  for  the 
development  of  a  simulator  evaluation  protocol  can  serve  as  a  reference  for  the  maritime 
community.  The  USCG  should  make  the  current  methodology  widely  available  to  the  industry 
by  publishing  this  report  as  a  public  domain  document,  and  by  encouraging  its  inclusion  in 
courses  on  performance-based  assessment  or  “train-the-trainer”  courses. 

Encourage  the  development  of  a  library  of  simulator  evaluation  protocols.  The  evaluation 
method  documented  in  this  report  should  be  applied  to  a  wide  range  of  simulators  so  as  to  create 
a  library  of  simulator  evaluation  protocols.  The  USCG  should  encourage  maritime  academies 
and  other  appropriate  institutions  to  apply  the  methodology  to  other  types  of  simulators  and  then 
share  general  lessons  learned,  model  protocols  for  other  types  of  simulators,  and  actual  results  of 
evaluations.  Examples  of  simulators  that  could  be  evaluated  using  the  present  approach  are 
ECDIS  simulators,  GMDSS  simulators,  vessel  traffic  system  simulators,  cargo  loading 
simulators,  and  diesel  engine  simulators. 

Standardize  simulator  evaluation  procedures  for  selected  types  of  simulators.  When  the 
methodology  is  better  understood  and  accepted,  the  USCG  should  develop,  or  encourage 
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appropriate  institutions  to  develop,  standardized  evaluation  procedures  for  various  types  of 
simulators.  These  procedures  could  include  standard  scenarios  and  conditions,  as  well  as 
guidelines  and  cut-off  scores  for  accepting  or  not  accepting  a  simulator  or  a  course  based  on  it. 

ARPA  Simulator  Evaluation  Protocol 

This  report  provides  an  example  of  applying  a  structured  methodology  in  developing  an  ARPA 
simulator  evaluation  protocol  based  on  mariner  assessment  requirements.  In  addition,  it  provides 
a  useful  approach  to  the  actual  evaluation  of  ARPA  simulators.  We  recommend  the  following 
actions  to  ensure  this  protocol  best  contributes  to  the  effectiveness  of  ARPA  simulator 
evaluations  in  the  maritime  industry. 

Publish  the  ARPA  simulator  evaluation  protocol  and  encourage  its  review  and  use.  The 

USCG  should  make  the  ARPA  simulator  evaluation  protocol  widely  available  and  encourage  its 
review  and  use  by  the  maritime  academies  and  other  appropriate  institutions,  such  as  public  and 
private  maritime  educational  institutions  and  shipping  companies. 

Use  the  ARPA  simulator  evaluation  protocol  in  the  assessment  of  simulator  requirements 
for  approval  of  training  courses.  With  greater  understanding  and  acceptance  of  the  approach, 
the  USCG  should  use  it  or  require  its  use  as  a  standard  evaluation  of  ARPA  simulators  for  use  in 
training  courses. 
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APPENDIX  A 


ARP  A  Simulator  Evaluation  Objectives,  Evaluation  Conditions,  and 
Evaluation  Criteria 

Appendix  A  provides  a  detailed  description  of  the  33  simulator  evaluation  objectives  we  derived 
from  the  performance-based  mariner  assessment  objectives  specified  in  McCallum  et  al.  (2000), 
STCW  standards  for  simulators  (IMO,  1996),  and  IMO  standards  for  actual  equipment  (IMO, 
1971,  1979).  The  appendix  is  divided  into  four  tables  (Tables  A-l  through  A-4),  corresponding 
to  the  four  evaluation  objective  categories:  exercise  programming,  equipment  set-up,  simulation, 
and  debriefing. 

Each  table  is  divided  into  eight  columns.  The  simulator  evaluation  objectives  are  described  in 
detail  in  the  first  column.  The  corresponding  mariner  assessment  objectives  from  McCallum  et 
al.  (2000)  are  listed  in  the  second  column.  In  the  third  column  the  evaluation  type  is  provided.  It 
is  either  static  or  dynamic,  depending  on  the  objective.  In  the  fourth  column  the  evaluation 
conditions  are  described,  and  the  exercise  used  to  evaluate  the  objective  is  specified. 

In  the  next  four  columns  the  evaluation  criteria  for  each  objective  are  described,  and  the 
reference  for  each  criterion  is  specified.  The  criteria  consist  of  the  controls  and  displays  that 
correspond  to  each  evaluation  objective.  For  example,  simulator  evaluation  objective  2.4,- 
Selection  of  safe  limits,  has  one  control  criterion,  2.4.C1,  Ability  to  select  safe  limits  according  to 
distance  (CPA)  and  time  (TCPA);  and  one  display  criterion,  2.4.  Dl,-  Indication  of  safe  limits. 
Both  of  these  criteria  are  required  for  an  ARPA  simulator  to  support  mariner  assessment 
objective  2.2,  Appreciation  of  the  uses,  benefits,  and  limitations  of  ARPA  operational  warnings. 
The  source  for  each  evaluation  criterion  is  indicated  in  the  reference  column.  Each  criterion  is 
derived  either  from  the  corresponding  mariner  assessment  objective  (MAO)  in  column  two;  IMO 
Resolution  A. 222  (VII),  Performance  standards  for  navigational  radar  equipment  (IMO,  1971); 
Resolution  A.422  (XI),  Performance  standards  for  automatic  radar  plotting  aids  (IMO,  1979); 
or  Section  A-I/12  of  the  amended  STCW,  Standards  governing  the  use  of  simulators  (IMO, 
1996). 
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Table  A-l.  ARPA  simulator  evaluation  objectives  in  the  exercise  programming  category. 
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Table  A-l.  ARPA  simulator  evaluation  objectives  in  the  exercise  programming  category.  (Continued) 
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Table  A-3.  ARPA  simulator  evaluation  objectives  in  the  simulation  category.  (Continued) 

Simulator  Mariner  Simulator  Evaluation  Criteria 

Evaluation  Assessment  Evaluation  Evaluation 
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APPENDIX  B 


ARPA  Simulator  Evaluation  Protocol 

This  simulator  evaluation  protocol  is  designed  to  address  PC-based  ARPA  simulators  with 
different  capabilities  to  determine  the  simulators’  utility  in  performance-based  assessment.  The 
mariner  assessment  objectives  addressed  in  this  simulator  evaluation  are  those  specified  in 
McCallum  et  al.  (2000).  The  protocol  has  three  sections: 

•  Simulator  Specifications.  This  section  includes  a  two-page  worksheet  for  recording 
simulator  specifications.  The  evaluator  uses  this  worksheet  to  identify  and  briefly  describe 
each  simulator. 

•  IMO  Scenarios.  This  section  consists  of  four  scenarios  specified  in  IMO  Resolution  A.422 
(XI),  Performance  standards  for  automatic  radar  plotting  aids  (IMO,  1979).  These 
scenarios  are  designed  to  test  the  simulator’s  ability  to  replicate  a  specific  ARPA  operational 
limitation,  the  time  delay  associated  with  inaccurate  sensor  inputs.  The  instructions  for  this 
section  are  on  page  B-4. 

•  Operational  Exercises.  This  section  contains  the  main  portion  of  the  evaluation  form.  It  is 
divided  into  four  sections  corresponding  to  exercises  A,  B,  E,  and  F  from  McCallum  et  al. 
(2000).  The  general  instructions  for  this  section  are  on  page  B-9;  instructions  for  the 
exercises  precede  the  evaluation  worksheets  for  each  exercise. 
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SIMULATOR  SPECIFICATIONS 


Date  of  Evaluation  _ Evaluator(s) _ 

Manufacturer  _ Model  _ _ 

Hard  Drive(s)  and  Monitor (s)  For  all  required  hard  drives,  describe  the  console  type  ( trainee 
or  instructor),  manufacturer,  processor,  bytes  of  RAM,  bytes  of  storage,  and  network  card.  For 
all  required  monitors,  describe  manufacturer,  size,  and  color  and  video  specifications. 

Test  Model: 

Console  Type  Hard  Drive  Monitor 

1.  _  _ _ _  _ 

2.  _  _  _ 

Minimum  Requirements: 

1.  _  _  _ 

2.  _  _  _ 

Operating  System  and  Source  Code 

Test  Model  _ Others  Available  _ _ 

Accessories  Check  all  that  are  available  and  circle  those  used  with  test  model. 

•  Keyboard  •  Mouse  •  Touchscreen  •  Trackball  •  Actual  radar  keyboard 

•  Other _ 

Radar  Interface  Check  all  that  are  available  and  circle  the  interface  used  on  test  model. 

•  Furuno _  •  Sperry _ . _ 

•  Kelvin  Hughes _  •  Military _ _ _ _ 

•  Racal  Decca _  •  Other(s) _ _ _ 

•  Raytheon _  •  No  specific  model 

Actual  Radar  Simulated 

Radar  Display  •  • 

Radar  Controls  •  • 

Algorithmic  Model  •  • 
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ARPA  Controls  Check  all  that  are  available,  and  circle  those  used  with  test  model. 


Trainee 

Instructor 

Icons 

• 

• 

Menus 

• 

• 

Function  keys 

• 

• 

Other 

• 

• 

Peripherals  Check  all  that  are  available ,  and  circle  those  used  with  test  model . 

•  Printer 

•  Plotter  •  ECDIS 

•  360°  sea  visualization  •  Ship  controls 

•  Other(s) 


Number  in 

Instructor 

Company  Engineers 

Library 

Can  Create 

Can  Create 

Land  Mass 

• 

• 

Own  Ship 

• 

• 

Target 

• 

• 

Number  of  target  ships  that  can  be  tracked  _ 

Exercise  Programming 

•  Pre-programmed  exercises  can  be  modified  •  Custom  exercises  can  be  programmed 

Number  of  pre-programmed  exercises  available  _ 

System  Troubleshooting 

•  Operating  manual  •  Demonstration  of  system  malfunctions 

Networking  Capability 

•  Instructor  console  •  Other  trainee  consoles  Number  of  trainee  consoles _ 

Cost  of  Test  Model 

Trainee  Console 

Hardware  _ 

Software  _ 

Total  $ _ 

B-3 


Instructor  Console  Complete  System 

$ _  $ _ 


IMO  SCENARIOS 


This  section  provides  the  means  to  determine  whether  the  simulator  can  achieve  the  performance 
standards  stipulated  in  four  standardized  scenarios  developed  by  IMO  (cited  in  Bole  &  Dineley, 
1990,  p.  364). 

Rationale 

The  level  of  accuracy  of  actual  ARPA  units  varies  depending  on  the  sensor  input  data  and 
technical  equipment  specifications.  For  this  reason,  IMO  has  specified  standards  that  should  be 
achieved  by  an  actual  ARPA  unit  under  four  specific  operational  conditions.  The  goal  of  this 
section  is  to  verify  that  the  simulator  replicates  the  operational  limitations  of  an  actual  ARPA  in 
each  of  these  four  conditions. 

Instructions 

The  ARPA  simulator  should  be  tested  on  the  four  standardized  IMO  scenarios  specified  in  the 
following  pages.  First,  the  scenario’s  initial  conditions  should  be  programmed  into  the  simulator. 
Each  scenario  includes  only  one  target  and  own  ship.  Second,  the  scenario  should  be  run  and  the 
target  steadily  tracked.  After  the  first  minute,  the  simulator’s  performance  data  should  be 
recorded.  The  tracking  should  be  continued,  and  after  three  total  minutes,  the  simulator’s 
performance  data  should  be  recorded  again.  After  the  scenario  ends,  the  entire  process  should  be 
repeated  five  more  times.  Repeating  the  scenarios  more  than  once  enables  evaluators  to  verify 
that  the  simulator  faithfully  replicates  the  limitations  of  actual  equipment.  Each  iteration  should 
generate  a  different  set  of  data. 
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These  data  represent  the  accuracy  values  after  three  minutes  of  steady  tracking.  The  TCPA,  true  course,  and  true  speed  should  not  be  available  for  the  first 
minute  of  steady  tracking. 


Program  the  data  in  Table  B-3  into  the  simulator  (relative  course  and  speed  are  provided  for  reference  only). 

Track  the  target  steadily  for  one  minute.  Then,  in  Table  B-4,  under  column  1  of  One-Minute  Accuracy  Trials,  record  the  relevant 
evaluation  criteria  data.  In  this  scenario,  there  should  not  be  a  CPA  or  TCPA.  If  the  simulator  generates  these,  note  them. 
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These  data  represent  the  accuracy  values  after  three  minutes  of  steady  tracking.  The  TCPA,  true  course,  and  true  speed  should  not  be  available  for  the  first 
minute  of  steady  tracking. 
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3  These  data  represent  the  accuracy  values  after  three  minutes  of  steady  tracking.  The  TCPA,  true  course,  and  true  speed  should  not  be  available  for  the  first 
minute  of  steady  tracking. 
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4  These  data  represent  the  accuracy  values  after  three  minutes  of  steady  tracking.  The  TCPA,  true  course,  and  true  speed  should  not  be  available  for  the  first 
minute  of  steady  tracking. 


OPERATIONAL  EXERCISES 


This  section  constitutes  the  core  of  the  simulator  evaluation  form.  We  organized  the  evaluation 
form  on  the  basis  of  four  of  the  operational  ARPA  exercises  specified  in  McCallum  et  al.  (2000). 
The  four  exercises  (A,  B,  E,  and  F)  begin  with  a  general  description  of  the  exercise  parameters, 
and  the  initial  data  (name,  bearing,  range,  course,  speed,  and  vessel  type)  for  all  targets  involved 
in  the  exercise.  Following  the  initial  target  data,  we  divided  each  exercise  into  at  most  four 
sections,  corresponding  to  the  simulator  evaluation  objective  categories:  exercise  programming, 
equipment  set-up,  simulation,  and  debriefing.  Each  exercise  has  simulation  objectives;  however, 
only  exercises  A,  B,  and  E  have  exercise  programming  objectives;  and  only  exercises  A,  B,  and 
F  have  equipment  set-up  objectives.  Exercise  B  is  the  only  one  that  has  debriefing  objectives. 

In  the  exercise  programming  section,  we  evaluated  each  simulator  on  its  ability  to  program  the 
initial  conditions  necessary  to  create  the  ARPA  exercise.  In  the  equipment  set-up  section,  we 
evaluated  each  simulator  on  its  ability  to  initialize  different  ARPA  features,  such  as  the  display 
orientation.  In  the  simulation  section,  we  evaluated  each  simulator  on  its  ability  to  simulate  the 
operational  capabilities  of  an  actual  ARPA  unit.  In  the  debriefing  section,  we  addressed  each 
simulator’s  ability  to  permit  an  instructor  to  monitor,  record,  replay,  and  print  exercises.  The 
following  data  are  addressed  on  the  evaluation  form: 

•  Time.  The  time  frame  for  the  exercise. 

•  Evaluation  Conditions.  The  required  actions  the  simulator  should  perform.  The  numbered 
conditions  address  specific  aspects  of  the  simulator  evaluation  objectives  listed  in 
Appendix  A.  The  non-numbered  conditions  are  described  in  the  cover  page  for  each 
exercise. 

•  Evaluation  Criteria.  Descriptions  of  the  control  and  display  features  to  be  evaluated. 

•  Availability.  An  indication  of  the  presence  or  absence  of  the  feature. 

•  Performance  Rating.  An  indication  of  how  well  the  simulator  performed  each  criterion. 

•  Comments.  A  record  of  any  other  pertinent  information  about  the  criterion. 

For  example,  the  equipment  set-up  section  of  exercise  A  contains  simulator  evaluation  objective 
2.1  -  Selection  of  display  presentation,  orientation,  and  vector  mode.  The  evaluation  conditions 
for  this  objective  specify  that  the  simulator  should  be  set  to  a  North-up  display  orientation.  The 
evaluation  criteria  for  objective  2.1  include  eleven  separate  controls  and  displays.  The  criteria 
relating  to  display  orientation  are  2.1.C2  -  Ability  to  toggle  between  North-up,  and  either  course- 
up  or  head-up  azimuth  stabilization;  and  2.1.D2  -  Indication  of  display  orientation.  When 
evaluating  these  criteria,  an  evaluator  would  indicate  in  the  availability  column  whether  the 
simulator  has  these  capabilities.  Then,  under  performance  ratings,  the  evaluator  would  indicate 
to  what  extent  the  simulator  met  the  criteria.  “Y”  (yes)  indicates  the  simulator  fully  meets  the 
criteria;  “P”  (partial)  indicates  the  simulator  partially  meets  the  criteria;  and  “N”  (no)  indicates 
the  simulator  does  not  meet  the  criteria. 
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Evaluation  Worksheets  for  ARP  A  Exercise  A 

Exercise  A  is  an  open  waters  scenario  with  2 1  ships  and  3  buoys,  including  6  container  ships  and 
15  fishing  vessels;  precipitation  clutter  is  present  near  the  buoys.  The  initial  data  for  the 
candidate’s  own  ship  and  the  target  ships  are  in  Table  B-9.  The  following  events  occur  in  this 
scenario: 

•  Own  ship  and  target  E  maneuver;  target  D  reduces  speed. 

•  Target  C  is  lost  (due  to  instructor  moving  target),  resulting  in  lost  target  alarm  actuation. 

•  The  speed  log  and  gyrocompass  are  disabled. 

•  Sea  clutter  appears  at  end  of  the  exercise  near  own  ship. 

These  events  are  noted  under  “Condition”  in  Table  B-12.  Refer  to  the  “Time”  column  of  that 
table  for  the  timing  of  each  event. 


Table  B-9.  Vessel  data  at  00:00  for  exercise  A  (McCallum  et  al.,  2000). 


Target  Name 

Bearing 

Range  nm 

Course 

Speed  kt 

Target  Type 

Own  Ship 

090° 

20 

Container 

A 

073° 

10.4 

D1W 

DIW 

Container 

B 

10.0 

090° 

7 

Container 

C 

131° 

10.6 

000° 

23 

Container 

D 

285° 

8.2 

090° 

25 

Container 

E 

050° 

10.0 

200° 

13 

Container 

F 

DIW 

DIW 

Buoy 

G 

DIW 

DIW 

Buoy 

H 

DIW 

DIW 

1 

DIW 

DIW 

J 

DIW 

K 

DIW 

L 

DIW 

DIW 

M 

DIW 

DIW 

N 

DIW 

DIW 

O 

DIW 

| 

P 

DIW 

i 

Q 

19 

DIW 

| 

R 

DIW 

S 

DIW 

T 

DIW 

Fishing 

U 

mm 

10 

Container 
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Instructions 

Program  the  data  in  Table  B-9  into  the  ARPA  simulator.  Review  the  exercise  programming 
conditions  given  in  column  2  of  Table  B-10.  Evaluate  the  simulator’s  ability  to  generate  the 
target  characteristics  and  location  required  for  this  exercise.  Next,  review  the  equipment  set-up 
conditions  in  Table  B-l  1 .  Determine  whether  the  simulator  can  be  set  up  per  the  required 
conditions.  Third,  review  the  simulation  conditions  in  Table  B-l 2.  Determine  whether  the 
simulator  can  simulate  the  events  in  a  realistic  and  dynamic  manner.  Under  “Availability,”  note 
whether  the  simulator  has  the  required  control  or  display.  Under  “Performance  ratings,”  indicate 
the  extent  to  which  the  simulator  satisfies  each  evaluation  criterion.  Y  (yes)  indicates  the 
simulator  fully  meets  the  criterion;  P  (partial)  indicates  the  simulator  partially  meets  the  criterion; 
and  N  (no)  indicates  the  simulator  does  not  meet  the  criterion.  If  limitations  are  found  for  any 
criterion,  detail  them  under  “Comments.” 


J 


B-l  1 
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Ability  to  program  ARPA 
_  failure 

1 .6  Reproduce  critical  Ability  to  program  density  and 
ARPA  operational  area  covered  by  sea  clutter 
limitations 


Table  B-ll.  Exercise  A  -  Evaluation  worksheet  for  equipment  set-up  criteria. 


Comments 

Performance 

Rating 

z 

CL 

> 

Avail¬ 

ability 

Y/N 

i 

Simulator  Evaluation  Criteria 

Display 

Indication  of  display  mode 

Indication  of  vector  mode 

Indication  of  display 
orientation 

Indication  of  range  scale  in 
use 

Indication  of  distance 
between  range  rings 

Indication  of  manual  speed 
input 

Indication  of  auto  speed  log 

o 

++ 

c 

o 

O 

Ability  to  toggle  between 
sea-  and  ground-stabilized 
modes 

Ability  to  toggle  between 
relative  and  true  motion 

Ability  to  toggle  among 
North-up,  course-up,  and 
head-up 

Ability  to  use  the  following 
ranges: 

(1)  12  or  16  mile 

(2)  3  or  4  mile 

Availability  of  fixed  range 
rings 

Availability  of  variable 
electronic  range  marker 

Ability  to  set  speed  log  with 

1  kt  resolution; 

(1)  manual 

(2)  automatic  speed  log 

Condition 

2.1  Display  mode  set 
to  sea-stabilized 

2.1  Vector  mode  set 
to  relative  motion 

2.1  Display 

orientation  set  to 
North-up 

2.1  Range  ring  set¬ 
up 

2.2  Speed  log  set  to 

20  knots 

Time 

Set-up 

Set-up 

Set-up 

Set-up 

Set-up 

B-13 
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Table  B-12.  Exercise  A  -  Evaluation  worksheet  for  simulation  criteria. 

(Continued) 


Comments 

Performance 

Rating 

z 

■ 

CL 

>- 

Avail¬ 

ability 

Y/N 

1 

Simulator  Evaluation  Criteria 

Display 

Target  data  (course,  speed, 
CPA,  TCPA,  range, 
bearing)  show  trend  after 
first  minute 

Target  data  are  more 
precise  after  three  minutes 

■ 

Visual  and/or  audible 
warning  when  target  is  lost 

Indication  of  last  tracked 
position 

o 

t. 

■«-» 

c 

o 

O 

Ability  to  activate  or 
deactivate  lost  target 
warning 

Lost  target  can  be 
reacquired 

Condition 

Own  ship  changes 
course  to  120° 

3.9  Display  of  target 
vectors/data 
following  own 
ship  course 
change 

Target  E  changes 
course  to  180° 

3.3  Loss  of  track  for 
target  C  and 
sounding  of  “lost 
target”  alarm 

Target  D  reduces 
speed  to  1 5  knots 

Time 

0106- 

0109 

0112 

0113 

0114 

B-17 
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log  error  warning 

Erroneous  own  ship  and 
target  speed  indications 


Table  B-12.  Exercise  A  -  Evaluation  worksheet  for  simulation  criteria. 

(Continued) 
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Evaluation  Worksheets  for  ARP  A  Exercise  B 

In  this  exercise,  the  candidate’s  own  ship  is  maneuvering  in  the  Strait  of  Gibraltar.  Three  vessel 
targets  (A,  B,  C)  proceed  in  various  directions,  while  nine  fishing  vessels  remain  in  fixed 
positions.  A  racon  (D)  is  on  the  coastline  at  Isla  Tarifa.  The  display  presentation  is  set  to  North- 
up.  This  scenario  features  other  programming  requirements  and  events: 

•  A  zone  of  shallow  depth  is  in  the  northeast  comer  of  the  exercise  area. 

•  A  radar  blind  sector  is  in  the  southwest  comer  of  the  exercise  area. 

•  Using  parallel  index  lines,  own  ship  maneuvers  to  270°  to  stay  3  nm  from  Isla  Tarifa. 


Table  B-13.  Vessel  data  at  00:00  for  exercise  B  (McCallum  et  al.,  2000). 


Target  Name 

Bearing 

Range  nm 

Course 

Speed  kt 

Type  of  Target 

Own  Ship 

255° 

20 

Container 

A 

230° 

4.8 

075° 

15 

Container 

B 

255° 

11.7 

090° 

16 

Container 

C 

255° 

6.0 

165° 

10 

Container 

D 

281° 

6.7 

DIW 

Racon 

E 

069° 

5.8 

255° 

30 

Container 

F 

207° 

5.8 

DIW 

Fishing 

G 

207° 

6.7 

Fishing 

H 

214° 

6.3 

DIW 

Fishing 

1 

207° 

4.3 

DIW 

Fishing 

J 

296° 

4.0 

DIW 

Fishing 

K 

296° 

3.0 

DIW 

Fishing 

L 

311° 

4.0 

DIW 

M 

312° 

3.0 

DIW 

N 

300° 

3.0 

DIW 

Instructions 

Program  the  data  in  Table  B-13  into  the  ARPA  simulator.  Review  the  exercise  programming 
conditions  given  in  column  2  of  Table  B-14.  Evaluate  the  simulator’s  ability  to  generate  the 
target  characteristics  and  location  required  for  this  exercise.  Next,  review  the  equipment  set-up 
conditions  in  Table  B-15.  Determine  whether  the  simulator  can  be  set  up  per  the  required 
conditions.  Third,  review  the  simulation  conditions  in  Table  B-16.  Determine  whether  the 
simulator  can  simulate  the  events  in  a  realistic  and  dynamic  manner.  Lastly,  review  the 
debriefing  conditions  in  Table  B-17.  Determine  whether  the  simulator  can  provide  the  required 
summaries  of  exercise  activities.  On  all  worksheets,  under  “Availability,”  note  whether  the 
simulator  has  the  required  control  or  display.  Under  “Performance  ratings,”  indicate  the  extent  to 
which  the  simulator  satisfies  each  evaluation  criterion.  If  limitations  are  found  for  any  criterion, 
detail  them  under  “Comments.” 
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B-21 


B-22 
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1 006  Evaluator  changes  own 
ship’s  course  to  270° 


Table  B-16.  Exercise  B  -  Evaluation  worksheet  for  simulation  criteria. 

(Continued) 


Comments 

Performance 

Rating 

z 

CL 

>• 

Avail¬ 

ability 

N/A 

Simulator  Characteristics  Evaluation  Criteria 

Display 

Indication  of  parallel 
index  lines  on  display 

Indication  of  false  echo 

Indication  of  radar 
interference 

Indication  of  blind  sector 

Control 

Ability  to  draw  parallel 
index  lines 

Availability  of  navigation 
lines  (optional) 

Ability  to  measure 
distance  between 
parallel  index  line  and 
land  mass 

Ability  to  measure 
bearing  of  parallel  index 
line 

Ability  to  measure  range 
between  own  ship  and 
parallel  index  line 

Ability  to  generate  false 
echo 

Ability  to  generate  radar 
interference  with  closest 
target 

Ability  to  simulate  blind 
sector 

i 

Conditions 

3.12  Use  of  parallel  index 
lines  to  maintain  3  nm 
at  bearing  180°  from 

Isla  Tarifa 

1 .6  Reproduce  false  echo 

1 .6  Reproduce  radar 
interference 

1.6  Reproduce  blind 
sector 

END  OF  SIMULATION 

Time 

1006 

fast 

fwd 

1018 

o 

C\J 

o 

o 

C\l 

O 
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Ability  to  print  Ability  to  print  screen 
screen  while  exercise  is  running 


Table  B-17.  Exercise  B  -  Evaluation  worksheet  for  debriefing  criteria. 

(Continued) 


Comments 

Performance 

Rating 

z 

Q- 

>- 

Avail¬ 

ability 

Y/N 

Simulator  Evaluation  Criteria 

Display 

Printout  of  exercise 
events: 

(1)  chart  view 

(2)  radar  view 

Ability  to  monitor  trainee 
station  using: 

(1)  chart  view 

(2)  radar  view 

Control 

Ability  to  print  exercise 
activities  following 
exercise 

Conditions 

4.3  Ability  to  print 
exercise 
activities 

4,4  Ability  to  monitor 
exercises 
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Evaluation  Worksheets  for  ARP  A  Exercise  E 

In  exercise  E,  the  candidate’s  own  ship  is  navigating  through  a  narrow  channel  in  New  York 
City’s  Upper  Harbor.  The  following  characteristics  are  present  in  this  scenario: 

•  A  cross  current  and/or  wind  is  present,  requiring  “crabbing”  of  own  ship  down  the  channel; 
crabbing  is  facilitated  by  the  use  of  ground-stabilized  mode. 

•  Own  ship  is  outbound. 

•  Target  A  is  inbound. 

•  Target  B  is  in  the  harbor,  dead  in  the  water. 


Table  B-18.  Vessel  data  at  00:00  for  exercise  E  (McCallum  et  al.,  2000). 


Target  Name 

Range  nm 

Bearing 

Course 

Speed  kt 

Target  Type 

Own  Ship 

180° 

15 

Container 

A 

7.0 

CD 

o 

347° 

11 

Container 

B 

4.0 

172° 

DIW 

Container 

Instructions 

Program  the  data  in  Table  B-18  into  the  ARPA  simulator.  Review  the  exercise  programming 
conditions  given  in  column  2  of  Table  B-19.  Evaluate  the  simulator’s  ability  to  generate  the 
current  and  wind.  Next,  review  the  simulation  conditions  in  Table  B-20.  Determine  whether  the 
simulator  can  be  set  up  per  the  requirements  of  this  scenario.  Evaluate  the  simulator’s  ability  to 
simulate  the  events  in  a  realistic  and  dynamic  manner.  Under  “Availability,”  note  whether  the 
simulator  has  the  required  control  or  display.  Under  “Performance  rating,”  indicate  the  extent  to 
which  the  simulator  satisfies  each  evaluation  criterion.  If  limitations  are  found  for  any  criterion, 
detail  them  under  “Comments.” 
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Evaluation  Worksheets  for  ARP  A  Exercise  F 


Exercise  F  is  an  open  waters  scenario  with  vessel  targets  A,  B,  and  C  starting  at  the  same  bearing 
(220°),  but  different  ranges  (1 1.5,  9.5,  and  7.5).  Targets  A  and  C  are  on  a  collision  course.  The 
situation  requires  the  candidate’s  own  ship  to  maneuver  to  starboard  to  keep  outside  of  the 
established  safe  limits  closest  point  of  approach  (CPA)  of  1  nm. 


Table  B-21.  Vessel  data  at  00:00  for  exercise  F  (McCallum  et  al.,  2000). 


Target  Name 

Range  nm 

Bearing 

Course 

Speed  kt 

Target  Type 

Own  Ship 

270° 

20.0 

Container 

A 

11.5 

220° 

005° 

26.5 

Container 

B 

9.5 

220° 

295° 

29.0 

Container 

C 

7.5 

220° 

335° 

16.8 

Container 

D 

6.0 

048° 

270° 

20.0 

Container 

Instructions 

Program  the  data  in  Table  B-21  into  the  ARPA  simulator.  Review  the  equipment  set-up 
conditions  given  in  column  2  of  Table  B-22.  Determine  whether  the  simulator  can  be  set  up  per 
the  required  conditions.  Then,  review  the  simulation  conditions  in  Table  B-23.  Evaluate  the 
simulator’s  ability  to  replicate  the  events  in  a  realistic  and  dynamic  manner.  To  evaluate  the 
accuracy  of  data  and  check  the  simulator’s  capability  to  generate  a  Search  and  Rescue 
Transponder  (SART),  repeat  the  simulation  portion  of  this  exercise,  using  the  worksheet 
provided  in  Table  B-24  to  record  the  results  of  the  second  iteration.  On  all  worksheets,  under 
“Availability,”  note  whether  the  simulator  has  the  required  control  or  display.  Under 
“Performance  ratings,”  indicate  the  extent  to  which  the  simulator  satisfies  each  evaluation 
criterion.  If  limitations  are  found  for  any  criterion,  detail  them  under  “Comments.” 
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APPENDIX  C 


Worksheets  for  Compiling  and  Analyzing  Simulator  Evaluation  Data 

This  appendix  provides  a  set  of  worksheets  for  compiling  and  analyzing  simulator  evaluation 
data.  The  appendix  is  divided  into  three  main  sections.  The  first  section  contains  four 
worksheets  (Tables  C-2  through  C-5),  on  which  the  evaluator  can  provide  a  detailed  summary  of 
the  findings  for  each  simulator  evaluation  criterion.  The  second  section  contains  Table  C-7,  a 
worksheet  that  the  evaluator  can  use  to  summarize  the  general  capability  of  a  simulator  to 
support  the  four  simulator  evaluation  objective  categories.  Lastly,  the  third  section  contains  a 
worksheet  (Table  C-9)  that  the  evaluator  can  use  to  indicate  a  simulator’s  capability  to  support 
the  current  set  of  mariner  assessment  objectives. 


WORKSHEETS  FOR  RECORDING  A  DETAILED  SUMMARY  OF  THE 
SIMULATOR  EVALUATION  RESULTS 


Tables  C-2,  C-3,  C-4,  and  C-5  are  worksheets  that  the  evaluator  can  use  to  summarize  the 
detailed  findings  of  a  simulator  evaluation.  The  four  tables  correspond  to  the  simulator 
evaluation  categories:  exercise  programming,  equipment  set-up,  simulation,  and  debriefing.  The 
tables  address  the  findings  for  all  the  individual  simulator  evaluation  criteria  within  each 
category.  As  noted  in  the  reference  column,  each  criterion  is  derived  either  from  the  present  set 
of  mariner  assessment  objectives  (MAO)  (McCallum  et  al.,  1999);  IMO  Resolution  A.222  (VII) 
(IMO,  1971);  Resolution  A.422  (XI)  (IMO,  1979);  or  Section  A-I/12  of  the  amended  STCW  Code 
(IMO,  1996).  In  the  next  section,  the  detailed  findings  for  each  evaluation  objective  are  tabulated 
and  presented  by  category  in  Table  C-7. 

In  the  present  application  to  ARPA  simulators,  we  used  a  subjective  ratings  approach 
(classification  into  three  or  more  pre-defined  levels)  to  score  each  criterion.  We  used  the 
following  definitions  for  each  level:  Y  (yes),  indicating  the  simulator  fully  satisfied  the  criterion; 
P  (partial),  indicating  the  simulator  partially  satisfied  the  criterion;  and  N  (no),  indicating  the 
simulator  did  not  satisfy  the  criterion.  Table  C-l  shows  an  example  of  how  we  used  the 
worksheet  depicted  in  Table  C-5  (debriefing)  to  summarize  our  evaluation  of  Simulator  Y  on 
simulator  evaluation  objectives  4. 1  through  4.4.  As  noted  in  the  “Rating”  column,  Simulator  Y 
was  capable  of  meeting  8  out  of  the  10  evaluation  criteria  noted  below.  Our  comments  note  the 
different  capabilities  of  the  simulator  with  respect  to  various  criteria. 


Table  C-l.  Detailed  summary  of  Simulator  T’s  ability  to  satisfy  debriefing  criteria. 


Simulator 

Evaluation 

Objectives 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

4.1  Record 
exercise 

4. 1  .C 1  Ability  to  specify  which  voyage 
parameters  to  display 

4. 1 .  D 1  Log  of  voyage  activities: 

(1)  own  ship  course  and  speed  at  a  given 
time 

(2)  target  ship  bearing,  range,  course, 
speed,  CPA  and  TCPA  at  a  given  time 

(3)  applicable  COLREGS  for  each  target 

(4)  operational  warnings 

STCW 

Y 

Y 

Log  of  activities  for 
(a)  and  (b)  are 
currently  available. 
Manufacturer 
indicated  that  logs 
for(c)  and  (d)  would 
be  available  soon. 

4.2  Replay 

4.2.C1  Ability  to: 

STCW 

Y 

exercise 

(1)  rewind 

(2)  fast  forward 

(3)  pause,  and 

(4)  save 

4.2.D1  Chart  view 

Y 

4.2.D2  Radar  view 

Y 
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Table  C-l.  Detailed  summary  of  Simulator  Y’s  ability  to  satisfy  debriefing  criteria. 

(Continued) 


Simulator 

Evaluation 

Objectives 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

4.3  Print  exercise 

4.3.C1 

Ability  to  print  screen  while  exercise 
is  running 

N 

4.3.D1 

Printout  of  screen 

N 

Not  available. 

4.3.C2 

Ability  to  print  a  hard  copy  of  exercise 
activities  in  different  views  following 

Y 

exercise 

4.3.D2 

Printout  of  exercise  events  in 
(a)  chart  view  (b)  radar  view 

Y 

Printer  was  not 
available  at  the  time 
of  evaluation. 

Printout  of  radar  view 
is  not  available. 

4.4  Monitor 

4.4.D1 

Ability  to  monitor  trainee  station 

STCW 

Y 

exercise 

using:  (a)  chart  view  (b)  radar  view 

Table  C-2.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
exercise  programming  criteria. 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

1.1  Create  an 
exercise  area 
that  enables 
the  simulation 
of  land 
masses 

1.1. C1 

1.1. D1.1 

1.1. D1.2 

Ability  to  generate  actual  coastline 
and  narrow  channel 

Indication  of  land  masses 

When  radar  antenna  is  mounted  at 

15m  the  equipment,  in  the  absence  of 
clutter,  gives  a  clear  indication  of  a 
coastline 

MAO 

MAO 

A.222 

1 .2  Replicate 

environmental 
conditions 
critical  to 

1.2. C1 

1.2. C2 

Ability  to  generate  depth 
characteristics 

Ability  to  generate  current 

MAO 

navigation  of 
own  ship 

1.2. C3 

1.2. C4 

Ability  to  generate  tidal  condition 

Ability  to  generate  wind 

MAO 

1.2.C5 

Ability  to  generate  precipitation  in 
specific  location  and  density 

1.2.D5.1 

Indication  of  precipitation  area 

1.2.D5.2 

Target  detection  is  influenced  by 
location  and  density  of  precipitation 

1.2.C6 

Ability  to  control  precipitation  clutter 

1.2.D6 

Indication  of  reduced  precipitation 
clutter 

C-3 


Table  C-2.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
exercise  programming  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

1 .3  Specify  own 

1.3. Cl 

Ability  to  select  model  of  ship 

MAO 

ship  parame¬ 
ters  to  create 

1.3.C2 

Ability  to  program  vessel  specifications 

realistic  navi- 

1.3.C3 

Ability  to  program  maneuvering 

gational  char- 

characteristics 

acteristics 

1.3.C4 

Ability  to  program  initial  set-up 
conditions 

1.3.D4 

Indication  of  own  ship  vector 

1.3.C5 

Ability  to  program  future  maneuvers 

MAO 

1.3.D6 

Effect  of  current  on  own  ship 
hydrodynamic  characteristics 

MAO 

1.3.D7 

Effect  of  current  on  own  ship  course 
and  speed 

1.3.D8 

Effect  of  wind  on  own  ship 
hydrodynamic  characteristics 

1.3.D9 

Effect  of  wind  on  own  ship  course  and 
speed 

1 .4  Specify  target 

1.4.C1 

Ability  to  program  minimum  of  20 

parameters  to 

targets 

create 

realistic 

1.4.C2 

Ability  to  select  model  of  ship 

navigational 

1.4.D2 

Visual  symbol  for  each  target  type 

scenario 

1.4.C3 

Ability  to  program  vessel  specifications 

MAO 

1.4.C4 

Ability  to  program  vessel  maneuvering 
characteristics 

1.4.C5 

Ability  to  select  racon,  buoy,  SART 

MAO 

1.4.D5 

Indication  of  target  code,  if  appropriate 

MAO 

1.4.C6 

Ability  to  obtain  range  and  bearing  of 
target 

A.422 

1.4.D6 

Indication  of  target  position 

1.4.C7 

Ability  to  program  initial  set-up 
conditions 

MAO 

1.4.C8 

Ability  to  program  future  maneuver 

MAO 

1.4.C9 

Ability  to  program  removal  or  addition 
of  target  from  the  exercise  area 

MAO 

1.4.C10 

Ability  to  program  target  fading  and 
replacement  with  SART 

1.4.D11 

Effect  of  current  on  target 
hydrodynamic  characteristics 

1 .4.D12  Effect  of  current  on  target  course  and 

speed 

1.4.D13  Effects  of  wind  on  target 

hydrodynamic  characteristics 

1.4.D14  Effect  of  wind  on  target  course  and 

speed 
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Table  C-2.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
exercise  programming  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

1.5  Reproduce 

1.5. Cl 

Ability  to  program  speed  log 

STCW 

critical 

malfunctions 

equipment 

malfunctions 

1.5.D1 

Indication  of  speed  log  malfunction 

1.5.C2 

Ability  to  program  gyrocompass 
malfunctions 

STCW 

1.5.D2.1 

Indication  of  gyrocompass  malfunction 

1.5.C3 

Ability  to  program  ARPA  failure 

STCW 

1.5.D3 

Indication  of  ARPA  malfunction 

1.6  Reproduce 

1.6. Cl 

Ability  to  program  density  and  area 

critical  ARPA 

covered  by  sea  clutter 

operational 

limitations 

1.6.D1 

Indication  of  sea  clutter  area 

III  1  1 1  LCUIUI  JO 

(i.e.,  effect  of 

1.6.C2 

Ability  to  control  sea  clutter 

limitations  on 
ARPA 

1.6.D2 

Indication  of  reduced  sea  clutter 

operations) 

1.6.C3 

Ability  to  generate  automatic  radar 
interference  with  closest  target  or  other 
ship 

1.6.C4 

Ability  to  generate  false  echo 

1.6.D4 

Indication  of  false  echo 

1.6.C5 

Ability  to  simulate  blind  sector 

1.6.D5 

Indication  of  areas  without  radar 
coverage 

Table  C-3.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
equipment  set-up  criteria. 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

2.1  Selection  of 

2.1. Cl 

Ability  to  toggle  between  sea-  and 

MAO 

display 

ground-stabilized  modes 

presentation, 

orientation, 

2.1. D1 

Indication  of  display  mode 

A. 422 

and  vector 

2.1. C2 

Ability  to  toggle  between  North-up, 

STCW 

mode 

and  either  course-up  or  head-up 
azimuth  stabilization 

A.422 

2.1.D2 

Indication  of  display  orientation  mode 

A.422 

2.1.C3 

Ability  to  toggle  between  relative  and 

STCW 

true  motion 

A.422 

2.1. D3 

Indication  of  display  vector  mode 

A.422 

2.1. C4 

Ability  to  use  ARPA  on  the  following 
ranges:  (a)  3  or  4  miles  and  (b)  12  or 

A.422 

16  miles 

2.1.C5 

Fixed  range  rings  available 

A.222 
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Table  C-3.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
equipment  set-up  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

2.1  .D5.1  Indication  of  range  scale  in  use 

2.1.D5.2  Indication  of  distance  between  range 
rings 

2. 1  .C6  Variable  electronic  range  marker 
available 

A.  422 

A.222 

A.222 

2.2  Selection  of 
required 
speed  and 
compass 
input 

2.2.  Cl  Ability  to  set  speed  log  input  with  1 

knot  resolution:  (a)  manual  (b) 
automatic 

2.2. D1.1  Indication  of  manual  speed  input 

2.2.  D1 .2  Indication  of  auto  speed  log 

2.2. C2  Ability  to  set  compass  log  input  with 

1°  resolution:  (a)  manual  (b) 
gyrocompass 

2.2. D2.1  Indication  of  manual  compass  input 

2.2. D2.2  Indication  of  gyrocompass  input 

STCW 

2.3  Selection  of 
ARPA  plotting 
controls  and 
manual/auto¬ 
matic 
acquisition 

2.3. C1  Ability  to  select  acquisition  rings  or 

areas 

2.3. D1  Indication  of  acquisition  rings  or 

areas 

2.3. C2  Ability  to  select  targets  and  initiate 

manual  target  acquisition 

2.3. D2  Indication  of  manual  acquisition  mode 

2.3. C3  Ability  to  select  targets  and  initiate 

automatic  target  acquisition 

2.3. D3  Indication  of  automatic  acquisition 

mode 

2.3. C4  Ability  to  select  target  manually  while 

in  automatic  acquisition 

STCW 

A.422 

A.422 

A.422 

A.422 

2.4  Selection  of 
safe  limits 

:  2.4.C1  Ability  to  select  safe  limits  according 
to  distance  (CPA)  and  time  (TCPA) 

2.4. D1  Indication  of  safe  limits 

STCW 

2.5  Selection  of 
vector  time 
scale 

2.5. C1  Ability  to  select  time-adjustable  or 

fixed  time  scale 

2.5. D1  Indication  of  time  scale  of  vector  in 

use 

STCW 

A.422 

A.422 

2.6  Selection  of 
exclusion 
areas  when 
automatic 
acquisition  is 
employed 

2. 6.  Cl  Ability  to  suppress  acquisition  in 

certain  areas  (i.e.,  to  select  exclusion 
area  according  to  bearing  and  range) 

2.6. D1  Indication  of  the  area  of  acquisition 

STCW 

A.422 

A.422 

2.7  Selection  of 
danger  area 

2.7. C1  Ability  to  create  a  danger  area 

2.7.  D1  Indication  of  danger  area 

MAO 
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Table  C-4.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
simulation  criteria. 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

3.1  Display 

3.1  .Cl 

Ability  to  toggle  between  presentation 

MAO 

characteristics 

modes:  (a)  ground-stabilized  (b)  sea- 

when 

stabilized 

alternating 

3.1.D1 

After  resetting  display  mode,  plotting 

A. 422 

between 

information  is  available  within  a  period 

ground-  and 

not  to  exceed  4  scans 

sea-stabilized 

modes 

3.1.C2 

Ability  to  toggle  between  display 
orientations:  (a)  course-up  (b)  head- 

up 

3.1. D2 

After  changing  display  orientation, 
plotting  information  is  available  within 
a  period  not  to  exceed  4  scans 

A.422 

3.2  Use  of 

3.2.C1 

Ability  to  acquire,  track,  process  and 

STCW 

manual  and 

continuously  update  information 

A.422 

automatic 

manually  for  at  least  10  targets 

acquisition 

3.2.D1.1 

Ability  to  display  information 
simultaneously  for  at  least  10  targets 
in  manual  mode 

A.422 

3.2.D1.2 

Indication  of  manually  tracked  targets 

A.422 

3.2.C2 

Ability  to  automatically  acquire,  track, 
process  and  continuously  update 
information  for  at  least  20  targets 

A.422 

3.2.D2.1 

Ability  to  display  information  for  at 
least  20  targets  simultaneously  in 
automatic  mode 

A.422 

3.2.D2.2 

Indication  of  automatically  tracked 
targets 

A.422 

3.2.C3 

Ability  to  suppress  automatic 
acquisition  mode 

A.422 

3.2.C4 

Ability  to  groundlock  a  target 

MAO 

3.2.D4 

Indication  of  groundlocked  target 

MAO 

3.3  Use  and 

3.3.C1 

Ability  to  activate  or  deactivate  "safe 

A.422 

limitations  of 

limit"  warning 

ARPA 

3.3.D1.1 

Visual  and/or  audible  warning  when 

A.422 

operational 

target  violates  safe  limit  criteria 

warnings 

3.3.D1.2 

Indication  of  target  causing  "safe  limit" 
warning 

A.422 

3.3.C2 

Ability  to  activate  or  deactivate  guard 
zone  warning 

A.422 

3.3.D2.1 

Visual  and/or  audible  warning  when 
target  enters  guard  zone  area 

A.422 

3.3.D2.2 

Indication  of  target  causing  “guard 
zone”  warning 

A.422 

3.3.C3 

Ability  to  activate  or  deactivate  "lost 
target"  warning 

A.422 

3.3.D3.1 

Visual  and/or  audible  warning  when 
target  is  lost 

A.422 

Table  C-4.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
simulation  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

3.3. D3.2  Indication  of  last  tracked  position 

3.3. C4  Lost  target  can  be  reacquired 

3.3. C5  Ability  to  activate  or  deactivate  "new 

target"  warning 

3.3. D5.1  Visual  and/or  audible  warning  when 

new  target  enters  the  acquisition  zone 

3.3. D5.2  Indication  of  target  causing  "new 

target"  warning 

3.3. C6  Ability  to  activate  or  deactivate  "target 

store  full"  warning 

3.3. D6  Visual  and/or  audible  "target  store  full" 

warning 

A.422 

A.422 

3.4  Detection  and 
identification 
of  false 
echoes,  sea 
returns, 
racons,  and 
SARTs 

3.4. C1  Ability  to  suppress  unwanted  echoes 

from  sea  clutter,  rain  and  other  types 
of  precipitation 

3.4. D1  Indication  of  reduced  precipitation 

clutter 

3.4. C2  Ability  to  adjust  the  sea  and 

precipitation  anti-clutter  manually  and 
continuously 

3.4. D2  Indication  of  reduced  sea  clutter 

3.4. C3  Ability  to  obtain  range  and  bearing  of 

any  object  on  display 

3.4. D3.1  Indication  of  racon  code 

3.4. D3.2  Indication  of  SART  code 

3.4. C4  Indication  of  target  fading  and 

replacement  with  SART 

STCW 

A. 222 

A. 222 

A.422 

3.5  Use  of 

graphic  repre¬ 
sentation  of 
danger  areas 

3.5.  D1  Indication  of  danger  areas 

3.5. D2  Visual  and/or  audible  warning 

3.6  Use  of  vessel 
history  trails 

3.6. C1  Ability  to  select  vessel  history  display 

3.6.  D1  Display  at  least  4  equally  time-spaced 

past  positions  of  any  targets  being 
tracked  over  a  period  of  at  least  8 
minutes 

3.6. D2  If  target  has  been  tracked  less  than  8 

minutes,  number  of  past  positions 
displayed  reflects  the  time  tracked 

3.6. D3  A  target  just  acquired  has  no  vessel 

history 

STCW 

A.422 

Table  C-4.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
simulation  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

3.7  Speed  and 

3.7.C1 

Ability  to  request  the  display  of  ARPA 

STCW 

direction  of  a 
target’s  rela- 

data 

A.422 

STCW 

tive  move- 

3.7.D1.1 

Acquired  data  (course,  speed,  CPA, 

ment  and  the 

TCPA,  range,  bearing)  show  trend  (low 

A.422 

identification 

accuracy)  for  first  minute 

of  critical 

3.7.D1.2 

Precise  target  data  (course,  speed, 

A.422 

echoes 

CPA,  TCPA,  range,  bearing)  appear 
after  three  minutes 

3.7.D1.3 

Acquired  target  data  (course  &  speed) 
should  be  displayed  in  a  vector  or 
graphic  form  which  indicates  the 
target’s  predicted  motion 

A.422 

3.7.D1.4 

ARPA  information  does  not  obscure 
radar  information 

A.422 

3.7.C2 

Ability  to  cancel  the  display  of 
unwanted  ARPA  data 

A.422 

3.8  Limitations  of 

3.8.D1 

Target  data  (course,  speed,  CPA, 

A.422 

vessel  data 

TCPA,  range,  bearing)  show  trend  (low 

following 

accuracy)  for  first  minute 

changes  in 

3.8.D2 

Precise  target  data  (course,  speed, 

A.422 

target’s  course 

CPA,  TCPA,  range,  bearing)  appear 

or  speed,  or 
both 

after  three  minutes 

3.9  Limitations  of 

3.9.D1 

Target  data  (course,  speed,  CPA, 

A.422 

vessel  data 

TCPA,  range,  bearing)  show  trend  (low 

following 
changes  in 
own  ship 
course,  speed, 
or  both 

3.9.D2 

accuracy)  for  first  minute 

Precise  target  data  (course,  speed, 

CPA,  TCPA,  range,  bearing)  appear 
after  three  minutes 

A.422 

3.10  Limitations  of 

3. 10. Cl 

Ability  to  program  numerous  targets  on 

MAO 

radar  range 
and  bearing 
on  the 
accuracy  of 

3.10.C2 

the  same  bearing 

Ability  to  auto-acquire  concurrently 
numerous  targets  on  same  bearing 

MAO 

ARPA  data 

3.10.D2 

Data  for  three  targets  on  same  bearing 
are  initially  less  accurate  than  data  for 

1  target 

3.11  The  circum¬ 

3.11.D1 

Visual  and/or  audible  "lost  target" 

stances 

warning 

causing 

3.11. D2 

Erroneous  indication  of  swapped 

"target  swap" 
and  their 

target’s  data 

effects  on 
display  data 

. 

C-9 


Table  C-4.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
simulation  criteria.  (Continued) 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

3.12  Use  of  parallel 
index  lines  to 
maintain 
position  on 
planned  course 
and  to  identify 
time  of 
maneuver 

3.12. C1  Ability  to  measure  distance  between 

parallel  index  line  and  land  mass 

3.12. D1  Ability  to  draw  parallel  index  lines 

maintaining  a  given  distance  from 
land 

3.12. C2  Ability  to  measure  bearing  of  parallel 

index  line 

3.12. D2  Indication  of  parallel  index  lines 

3.12. C3  Ability  to  measure  range  between 

own  ship  and  parallel  index  line. 

3.12. C4  Availability  of  navigation  lines 

(optional) 

MAO 

MAO 

MAO 

MAO 

3.13  Display  char¬ 
acteristics 
when  alternat¬ 
ing  between 
true  and  rela¬ 
tive  vectors 

3.13. C1  Ability  to  switch  between  vector 

modes  without  losing  tracking 
information 

3.13. D1  Indication  of  vector  mode 

A.422 

3.14  The  operation 
of  the  trial 
maneuver 
facility 

3.14. C1  Simulation  is  initiated  by  depression 

of  either  a  spring-loaded  switch,  or  a 
function  key 

3.14. D1  Identification  of  trial  maneuver  mode 

3. 1 4. C2  Ability  to  use  a  static  or  dynamic 

display 

3.14. D2  Simulate  the  effect  on  all  tracked 

targets  of  an  own  ship  maneuver 

3. 1 4. C3  Ability  to  include  a  time  delay 

3. 1 4. D3  Simulate  without  interrupting  the 

update  of  target  information 

STOW 

A.422 

A.422 

A.422 

MAO 

A.422 

3.15  Performance 
checks  of 
radar, 
compass, 
speed  input 
sensors,  and 
ARPA 

3.15. C1  Ability  to  reset  disabled  speed  Jog 

1  3.15.D1.1  Visual  and/or  audible  warning  "speed 
log  error" 

3.15. D1 .2  Erroneous  own  ship  and  target 

speed  and  course  indications 

3.15. C2  Ability  to  reset  disabled  compass  log 

3.15. D2.1  Visual  and/or  "compass  log  error" 

warning 

3.15. D2.2  Erroneous  own  ship  and  target 

speed  and  course  indications 

3.15. D2.3  Only  head-up  relative  motion  display 

is  available  when  gyrocompass  fails 

STOW 

STOW 

3.16  Methods  of 
testing  for 
malfunctions  of 
ARPA  systems 
including 
functional  self¬ 
testing 

3.1 6.C1  Test  programs  are  available  to 

assess  ARPA’s  overall  performance 
against  a  known  solution 

STCW 

A.422 
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Table  C-5.  Worksheet  for  recording  a  detailed  summary  of  a  simulator’s  ability  to  satisfy 
debriefing  criteria. 


Simulator 

Evaluation 

Objective 

Simulator  Evaluation  Criteria 

Reference 

Rating 

Comments 

4.1  Record 
exercise 

4. 1  .Cl  Ability  to  specify  which  voyage 
parameters  to  display 

4.1  .D1  Log  of  voyage  activities: 

(5)  own  ship  course  and  speed  at  a  given 
time 

(6)  target  ship  bearing,  range,  course, 
speed,  CPA  and  TCPA  at  a  given 
time 

(7)  applicable  COLREGS  for  each  target 

(8)  operational  warnings 

STCW 

4.2  Replay 
exercise 

4.2.  Cl  Ability  to: 

(1)  rewind 

(2)  fast  forward 

(3)  pause 

(4)  save 

4.2. D1  Chart  view 

4.2. D2  Radar  view 

STCW 

4.3  Print  exercise 

4.3. C1  Ability  to  print  screen  while  exercise 

is  running 

4.3. D1  Printout  of  screen 

4.3. C2  Ability  to  print  a  hard  copy  of  exercise 

activities  in  different  views  following 
exercise 

4.3. D2  Printout  of  exercise  events  (a)  chart 

view  (b)  radar  view 

4.4  Monitor 
exercise 

4.4.D1  Ability  to  monitor  trainee  station 

using:  (a)  chart  view  (b)  radar  view 

STCW 

WORKSHEET  FOR  SUMMARIZING  A  SIMULATOR’S  GENERAL 
CAPABILITY  TO  MEET  SIMULATOR  EVALUATION  OBJECTIVES 


This  section  contains  a  worksheet  that  the  evaluator  can  use  to  record  a  simulator’s  capabilities  in 
the  following  general  categories:  exercise  programming,  equipment  set-up,  simulation,  and 
debriefing.  Since  the  evaluation  of  an  ARPA  simulator  using  the  present  approach  can  be  quite 
extensive  (over  170  evaluation  criteria),  this  worksheet  (Table  C-7)  should  be  used  to  record  only 
the  general  findings.  The  general  findings  for  each  evaluation  objective  are  the  result  of 
tabulating  the  detailed  findings  for  each  criterion  recorded  in  Tables  C-2  through  C-5.  For 
example,  in  our  evaluation  of  the  debriefing  capabilities  of  ARPA  Simulator  Y  (see  Table  C-l) 
we  found  that  Y  could  replicate  8  out  of  10  of  the  evaluation  criteria  required  for  debriefing. 

Table  C-6  shows  how  these  results  were  recorded  onto  a  general  summary  worksheet. 


Table  C-6.  General  summary  of  Simulator  T’s  ability  to  satisfy  simulator  evaluation 
objectives  in  the  debriefing  category. 


Simulator  Evaluation  Objective 

Evaluation 
Criteria  Met 

Comments 

4. 

Debriefing 

80% 

4.1 

Record  exercise 

2/2 

Each  exercise  can  be  recorded  and  kept  in 
memory  for  an  extensive  period  of  time. 

4.2 

Replay  exercise 

3/3 

Each  exercise  can  be  replayed  in  real  and  fast 
time  using  either  the  radar  view  (instructor  or 
trainee  console)  or  the  chart  view  (instructor 
console). 

4.3 

Print  exercise 

2/4 

Manufacturer  indicated  that  all  the  logs  of 
voyage  activities,  as  well  as  chart  and  radar 
views  of  the  exercises,  can  be  printed. 

Printer  was  not  available  during  the  evaluation. 

‘Print  screen’  feature  is  not  available. 

4.4 

Monitor  exercise 

1/1 

Instructor  can  monitor  the  exercise  using  either 
the  chart  or  radar  view. 

To  obtain  these  results,  we  first  integrated  the  detailed  findings  (addressed  in  Table  C-l)  to 
determine  the  extent  that  Simulator  Y  met  the  debriefing  objectives.  Then,  we  assigned  a 
numerical  score  [1  (yes),  0.5  (partial)  and  0  (no)]  to  each  criterion.  Next,  we  summed  these 
scores  across  all  the  evaluation  criteria  within  each  simulator  evaluation  objective  to  obtain  a 
final  score  for  each  objective.  In  the  column  labeled  “evaluation  criteria  met,”  we  recorded  the 
ratio  of  the  criteria  met  to  the  total  criteria  available  for  each  objective.  For  example,  simulator 
evaluation  objective  4.3,  Print  exercise,  has  four  separate  evaluation  criteria  (see  Table  C-l  for  a 
listing  of  these  individual  criteria).  Simulator  Y  fully  met  two  criteria  (2  points)  and  did  not  meet  «. 
two  other  criteria  (0  points)  for  a  total  score  of  2  out  of  4  points.  This  score  is  recorded  above  as 
“2/4.”  Simulator  7’s  overall  percentage  score  (80%)  is  also  recorded  on  this  worksheet,  as  are 
our  comments  addressing  the  simulator’s  strengths  and  weaknesses  with  respect  to  each 
evaluation  objective.  Evaluators  can  also  use  the  comments  section  to  note  those  objectives  they 
were  unable  to  evaluate  due  to  constraints  of  the  exercises  used  in  the  evaluation. 
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Table  C-7.  Worksheet  for  summarizing  findings  of  an  ARPA  simulator  evaluation, 
organized  by  simulator  evaluation  objective. 


Simulator  Evaluation  Objective 

Evaluation 

Criteria  Met 5 

Comments 

1.  Exercise  Programming 

% 

1.1  Create  an  exercise  area  that  enables 
the  simulation  of  land  masses 

/3 

1 .2  Replicate  environmental  conditions 
critical  to  navigation  of  own  ship 

/9 

1 .3  Specify  own  ship  parameters  to 
create  realistic  navigational 
characteristics 

/10 

1 .4  Specify  target  parameters  to  create 
realistic  navigational  scenario 

/1 7 

1 .5  Reproduce  critical  equipment 
malfunctions 

/6 

1 .6  Reproduce  critical  ARPA  operational 
limitations  (i.e.,  effects  of  limitations 
of  ARPA  operations) 

/9 

2.  Equipment  Set-Up 

% 

2.1  Selection  of  display  presentation, 
orientation,  and  vector  mode 

/II 

2.2  Selection  of  required  speed  and 
compass  input 

16 

2.3  Selection  of  ARPA  plotting  controls 
and  manual/automatic  acquisition 

17 

2.4  Selection  of  safe  limits 

12 

2.5  Selection  of  vector  time  scale 

12 

2.6  Selection  of  exclusion  areas  when 
automatic  acquisition  is  employed 

12 

2.7  Selection  of  danger  area 

12 

3.  Simulation 

% 

3.1  Display  characteristics  when 

alternating  between  ground-  and  sea- 
stabilized  modes 

/4 

3.2  Use  of  manual  and  automatic 
acquisition 

/9 

3.3  Use  and  limitations  of  ARPA 
operational  warnings 

/1 5 

3.4  Detection  and  identification  of  false 
echoes,  sea  returns,  racons,  and 
search  and  rescue  transponders 
(SART) 

/8 

3.5  Use  of  graphic  representation  of 
danger  areas 

12 

3.6  Use  of  vessel  history  trails 

14 

5  The  denominator  represents  the  total  number  of  evaluation  criteria  for  each  evaluation  objective.  (See  Tables  C-2  through  C-5 
for  a  complete  listing  of  the  simulator  evaluation  criteria  for  each  objective.) 


Table  C-7.  Worksheet  for  summarizing  findings  of  an  ARPA  simulator  evaluation, 
organized  by  simulator  evaluation  objective.  (Continued) 


Simulator  Evaluation  Objective 

Evaluation 

Criteria  Met 

Comments 

3.7  Speed  and  direction  of  a  target's 
relative  movement  and  the 
identification  of  critical  echoes 

16 

3.8  Limitations  of  vessel  data  following 
changes  in  target  course  or  speed,  or 
both 

12 

3.9  Limitations  of  vessel  data  following 
changes  in  own  ship  course,  speed, 
or  both 

12 

3.10  Limitations  of  radar  range  and 
bearing  on  the  accuracy  of  ARPA 
data 

13 

3.1 1  The  circumstances  causing  "target 
swap"  and  their  effects  on  display 
data 

12 

3.12  Use  of  parallel  index  lines  to  maintain 
position  on  planned  course  and  to 
identify  time  of  maneuver 

16 

3.13  Display  characteristics  when 

alternating  between  true  and  relative 
vectors 

12 

3.14  The  operation  of  the  trial  maneuver 
facility 

16 

3.15  Performance  checks  of  radar, 

compass,  speed  input  sensors,  and 
ARPA 

17 

3.16  Methods  of  testing  for  malfunctions  of 
ARPA  systems  including  functional 
self-testing 

/I 

4.  Debriefing 

% 

4.1  Record  exercise 

12 

4.2  Replay  exercise 

13 

4.3  Print  exercise 

14 

4.4  Monitor  exercise 

/i 

V 


f 
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WORKSHEET  FOR  SUMMARIZING  A  SIMULATOR’S  CAPABILITY  TO 
SUPPORT  MARINER  ASSESSMENT  OBJECTIVES 


This  section  contains  a  worksheet  (Table  C-9)  on  which  evaluators  can  record  a  simulator’s 
capability  to  support  the  27  mariner  assessment  objectives  described  in  McCallum  et  al.  (1999). 
Table  C-9  summarizes  a  simulator’s  performance  in  terms  of  its  ability  to  reproduce  the 
assessment  conditions  required  for  each  mariner  assessment  objective.  To  determine  the 
percentage  of  assessment  objectives  that  a  simulator  could  support,  the  evaluator  can  assign  a 
numerical  score  of  1  (yes),  0.5  (partial),  and  0  (no)  to  each  assessment  objective,  and  sum  these 
scores  across  objectives  within  an  assessment  objective  category.  Then,  the  evaluator  can  divide 
the  result  by  the  total  score  possible  for  that  category.  For  example.  Table  C-8  below  shows  an 
excerpt  of  our  summary  of  the  ability  of  Simulator  Y  to  support  mariner  assessment  objectives  in 
category  1,  setting  up  and  maintaining  displays.  In  this  category.  Simulator  Y  fully  supported 
four  assessment  objectives  (4  points)  and  partially  supported  two  objectives  (1  point),  resulting  in 
a  total  score  of  5  out  of  6  possible  points,  or  83  percent. 


Table  C-8.  Summary  of  the  capability  of  Simulator  Y  to  support  mariner  assessment 
objectives  in  category  1,  setting  up  and  maintaining  displays. 


Mariner  Assessment  Objective 

Simulator 

Evaluation 

Objective 

Ability  to  Meet 
Requirements 

Comments 

1.  Setting  up  and  maintaining  displays 

83% 

1.1 

The  selection  of  display  presentation; 
stabilized  relative  motion  display  and 
true  motion  display 

2.1 

Yes 

1.2 

The  selection,  as  appropriate,  of 
required  speed  and  compass  input  to 
ARPA 

2.2 

Yes 

Manual  input  as  well  as  automatic 
speed  log  and  gyrocompass  are 
available.  Resolution  is  0.1  knot 
or  degree. 

1.3 

The  selection  of  ARPA  plotting 
controls,  manual/automatic 
acquisition,  vector/graphic  display  of 
data 

2.1,  2.3 

Yes 

1.4 

The  selection  of  the  vector  time  scale 

2.5 

Yes 

1.5 

The  use  of  exclusion  areas  when 
automatic  acquisition  is  employed  by 
ARPA 

2.6 

Partial 

Although  exclusion  areas  are  not 
directly  available,  they  can  be 
indirectly  specified  using  two 
acquisition  areas  or  set  of  rings. 

1.6 

Display  characteristics  and  an 
understanding  of  when  to  use 
ground-  or  sea-stabilized  modes 

1.1,  1.2, 

3.1 

Partial 

Although  ground-stabilized  mode 
is  not  available,  the  availability  of 
randomly  generated  currents, 
along  with  the  use  of  EBL  and 

VRM,  enables  the  trainee  to 
understand  the  effect  of  current  on 
navigation. 

Table  C-9.  Worksheet  for  recording  a  summary  of  a  simulator’s  capability  to  support 
mariner  assessment  objectives. 


Mariner  Assessment  Objective 

Simulator 

Evaluation 

Objective 

Ability  to  Meet 
Requirements 

Comments 

1.  Setting  up  and  maintaining  displays 

% 

1 . 1  The  selection  of  display  presentation; 
stabilized  relative  motion  display  and 
true  motion  display 

2.1 

t  .2  The  selection,  as  appropriate,  of 

required  speed  and  compass  input  to 
ARPA 

2.2 

1 .3  The  selection  of  ARPA  plotting 
controls,  manual/automatic 
acquisition,  vector/graphic  display  of 
data 

2.1,  2.3 

1 .4  The  selection  of  the  vector  time  scale 

2.5 

1 .5  The  use  of  exclusion  areas  when 
automatic  acquisition  is  employed 

2.6 

1 .6  Display  characteristics  and  an 

understanding  of  when  to  use  ground 
or  sea-stabilized  modes 

1.1,  12, 

3.1 

2.  Situation  assessment 

% 

2. 1  Understanding  the  criteria  for  the 
selection  of  targets  by  automatic 
acquisition 

3.2 

2.2  Uses,  benefits  and  limitations  of 

ARPA  operational  warnings 

2.4,  3.3 

2.3  Detection  and  identification  of  false 
echoes,  sea  return,  racons,  and 

SART 

3.4 

2.4  The  use  of  graphic  representation  of 
danger  areas 

2.7,  3.5 

2.5  Knowledge  and  recognition  of  historic 
data  as  a  means  of  indicating  recent 
maneuvering  of  targets 

3.6 

2.6  The  speed  and  direction  of  a  target’s 
relative  movement  and  the 
identification  of  critical  echoes  (in 
both  relative  and  true  motion  modes 
of  display) 

3.7 

2.7  Detecting  target  course  and  speed 
changes  and  the  limitations  of  such 
information  (in  both  relative  and  true 
motion  modes  of  display) 

3.8 

2.8  The  effect  of  changes  in  own  ship’s 
course  or  speed  or  both  (in  both 
relative  and  true  motion  modes  of 
display) 

3.9 
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Table  C-9.  Worksheet  for  recording  a  summary  of  a  simulator’s  capability  to  support 
mariner  assessment  objectives.  (Continued) 


Mariner  Assessment  Objective 

Simulator 

Evaluation 

Objective 

Ability  to  Meet 
Requirements 

Comments 

3.  Knowledge  of  factors  affecting 
performance  and  accuracy;  and  ability 
to  operate  and  interpret  system 
performance  and  accuracy,  tracking 
capabilities  and  limitations,  and 
processing  delays 

% 

3.1  Knowledge  of  the  effect  of  limitations 
of  radar  range  and  bearing  on  the 
accuracy  of  ARPA  data 

3.10 

3.2  The  circumstances  causing  Target 
swap”  and  their  effect  on  display  data 

3.11 

3.3  The  effects  on  tracking  of  “lost” 
targets  and  target  fading 

3.4  An  appreciation  of  the  IMO  perform¬ 
ance  standards  for  ARPA,  in  particu¬ 
lar  the  standards  relating  to  accuracy 

3.3 

1.6,  3.15 
IMO 

exercises 

4.  Parallel  indexing 

% 

4.1  Plotting  parallel  index  lines  to 
maintain  position  on  planned  course 

4.2  Using  parallel  index  lines  to  identify 
time  of  maneuver 

1.1,3.12 

3.12 

5.  Application  of  COLREGS;  and 
deriving  and  analyzing  information, 
critical  echoes,  exclusion  areas  and 
trial  maneuvers 

% 

5.1  The  benefit  of  switching  between  true 
and  relative  vectors 

3.13 

5.2  Analysis  of  potential  collision 

situations  from  displayed  information, 
determination  and  execution  of  action 
to  avoid  close-quarters  situations  in 
accordance  with  COLREGS 

3.7 

5.3  The  operation  of  the  trial  maneuver 
facility 

3.14 

6.  Use  of  operational  warnings  and 
system  tests 

% 

6.1  Performance  checks  of  radar, 

compass,  speed  input  sensors  and 
ARPA 

1.5,  3.15 

6.2  Methods  of  testing  for  malfunctions  of 
ARPA  systems  including  functional 
self-testing 

1.5,  3.16 

6.3  Precautions  to  be  taken  after  a 
malfunction  occurs 

3.15 

6.4  Ability  to  perform  system  checks  and 
determine  data  accuracy  of  ARPA, 
including  the  trial  maneuver  facility, 
by  checking  against  basic  radar  plot 

1.5,  1.6, 
3.14,  3.16 
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